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Executive  Summary 


This  document  summarizes  the  work  done  by  the  Helix  team  in  2017  and  highlights  the  work 
planned  in  2018.  In  particular,  it  provides  details  on: 

•  Additional  analyses  conducted  by  the  Helix  team 

•  Additional  data  collected  by  the  Helix  team 

•  The  change  in  research  questions,  which  are  now: 

o  How  can  organizations  improve  the  effectiveness  of  their  systems  engineering 
workforce? 

o  How  does  the  effectiveness  of  the  systems  engineering  workforce  impact  the 
overall  systems  engineering  capability  of  an  organization? 

o  What  critical  factors,  in  additional  to  workforce  effectiveness,  are  required  to 
enable  systems  engineering  capability? 

•  The  three  companion  documents  to  this  report  developed  by  the  Helix  team: 

o  Atlas  1.1:  The  Theory  of  Effective  Systems  Engineers  -  (SERC-2018-TR-101-A) 
This  is  an  incremental  evolution  of  Atlas  that  reflects  feedback  from  the 
community,  additional  analysis,  and  maturation  of  the  team's  thinking  in  2017. 
In  particular.  Atlas  includes  minor  updates  on  the  values  systems  engineers 
provide,  the  roles  systems  engineers  play,  the  proficiency  model  for  systems 
engineers,  and  the  personal  characteristics  of  systems  engineers.  Henceforth, 
this  will  be  referred  to  as  "Atlas  1.1". 

o  Atlas  Career  Path  Guidebook  -  (SERC-2018-TR-101-B)  This  document  provides 
analyses  of  the  Helix  dataset,  providing  common  patterns  in  systems  engineers' 
careers.  The  Guidebook  also  provides  some  insights  on  questions  commonly 
asked  of  the  Helix  team  around  career  paths  and  the  team's  responses.  Finally, 
additional  work  on  linking  proficiencies  to  career  paths  has  been  completed  and 
is  reflected  in  the  guide.  Henceforth,  this  will  be  referred  to  as  the  "Guidebook” . 

o  Atlas  1.1.  Implementation  Guide:  Moving  from  Theory  Into  Practice  -  (SERC- 
2018-TR-101-C)  Whenever  Atlas  is  presented,  there  are  many  questions  about 
how  to  take  the  theory  and  apply  it  in  practice.  The  Guide  provides  examples 
from  organizations  that  have  implemented  parts  of  Atlas,  and  guidance  created 
by  the  Helix  team  based  on  many  interactions  with  organizations  around 
implementation  as  well  as  the  extensive  Helix  dataset.  Henceforth,  this  will  be 
referred  to  as  the  "Implementation  Guide". 

The  future  vision  for  Atlas  includes  developing  a  theory  of  systems  engineering  capability  which 
is  predicated  on  the  hypothesis  that  an  appropriately  skilled  systems  engineering  workforce 
supported  by  culture,  governance,  and  infrastructure  will  deliver  an  effective  systems 
engineering  capability. 
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1  Background  and  Introduction 


The  Systems  Engineering  Research  Center  (SERC),  a  University  Affiliated  Research  Center 
(UARC),  set  up  by  the  U.S.  Department  of  Defense  (DoD),  responded  to  the  systems  engineering 
workforce  challenges  by  initiating  the  Helix  Project  to  investigate  the  "DNA"  of  systems 
engineers,  beginning  with  those  who  work  in  defense  and  then  more  broadly.  The  US  Deputy 
Assistant  Secretary  of  Defense  for  Systems  Engineering  (DASD(SE)),  the  International  Council  on 
Systems  Engineering  (INCOSE)  and  the  Systems  Engineering  Division  of  the  National  Defense 
Industrial  Association  (NDIA-SED)  jointly  sponsor  Helix.  To  ensure  Helix  delivers  the  greatest 
value  and  to  help  Helix  obtain  access  to  the  necessary  data.  Helix  formed  the  Helix  Advisory 
Panel  (HAP)  with  representatives  primarily  from  those  three  sponsor  organizations.  Helix  has 
held  three  annual  workshops  with  a  broad  set  of  representatives  from  across  government, 
academia,  and  industry. 

Helix  is  a  multi-year  longitudinal  research  project,  which  has  gathered  data  from  many 
organizations  with  DoD  and  the  Defense  Industrial  Base  (DIB)  through  a  combination  of 
techniques,  including  interviews  with  hundreds  of  systems  engineers.  In  2014,  Helix  began  to 
reach  beyond  DoD  and  the  DIB,  to  gather  data  from  other  types  of  organizations  as  well, 
including  non-defense  organizations  in  the  US  and  non-US  organizations.  Version  0.25  of  Atlas 
was  also  published  in  2014.  Atlas  identifies  the  key  variables  that  impact  a  systems  engineer's 
effectiveness  -  positively  or  negatively  -  and  provides,  as  much  as  possible,  details  on  how 
these  variables  impact  effectiveness. 

During  2015,  Helix  expanded  its  data  collection  by  conducting  interviews  with  non-DoD 
organizations  as  well;  matured  Atlas  into  the  next  versions.  Atlas  0.6;  defined  and  analyzed  the 
career  paths  of  systems  engineers;  and  did  implementation  trials  of  Atlas. 

During  2016,  the  team  generated  Atlas  0.6  and  Atlas  1.0.  Atlas  1.0  reflects  the  results  of 
analysis  of  in-depth  interviews  with  287  individuals.  Most  of  these  individuals  were  systems 
engineers,  though  approximately  10%  of  the  sample  was  comprised  of  individuals  who  work 
with  systems  engineers  -  organizational  leaders,  classic  engineers  (electrical,  mechanical, 
software,  etc.),  and  program  managers.  In  2016  the  Helix  team  also  worked  on  implementation 
of  Atlas  with  a  number  of  organizations  and  lessons  learned  from  those  activities  are  captured 
here. 


1.1  The  Helix  Project 

The  US  Department  of  Defense  (DoD)  and  the  Defense  Industrial  Base  (DIB)  -  contractors  that 
develop  and  deliver  systems  to  the  DoD  -  have  been  facing  major  systems  engineering 
challenges  in  recent  years  (e.g.  GAO  2008,  2011,  2012,  2013).  Mission  requirements  are 
evolving  and  they  demand  ever  more  sophisticated  and  complex  systems  (e.g.  Boehm  et  al. 
2010;  INCOSE  Technical  Operations  2007;  Davidz  2006;  Davidz  and  Nightingale  2007;  Frank  et 
al.  2007;  INCOSE  2014);  the  tools,  processes,  and  technologies  that  systems  engineers  must 
master  keep  changing  more  rapidly  (e.g.  Frank  2006);  and  budgets  and  schedules  are  being 
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compressed  dramatically.  An  additional  concern  is  that  thousands  of  systems  engineers  in  the 
defense  workforce  are  nearing  retirement;  they  will  take  with  them  hundreds  of  thousands  of 
staff-years  of  experience  (DoD  2013). 

Organizations  have  responded  to  these  challenges  in  a  variety  of  ways,  such  as  offering 
extended  training  and  education  to  their  current  workforce  or  systematically  seeking  to  select 
specialty  engineers  with  the  potential  to  become  systems  engineers  and  incorporating  them 
into  the  ranks  of  systems  engineers.  Unknown  is  whether  these  actions  are  producing  the 
desired  results  because  there  is  no  common  understanding  of  the  diverse  roles  that  systems 
engineers  play,  how  they  are  selected  and  evaluated,  what  competencies  are  most  important 
for  different  roles,  how  to  evaluate  effectiveness,  or  how  experiences  impact  effectiveness. 
These  and  many  other  insights  will  be  critical  to  maintaining  and  growing  the  systems 
engineering  workforce  in  the  US  DoD  and  DIB. 


1.2  How  is  Atlas  Different  from  Helix? 

Helix  is  the  name  of  the  overarching  SERC  project.  Helix  has  been  examining  what  makes 
systems  engineers  effective  for  over  four  years.  As  a  project.  Helix  has  created  many  different 
deliverables  or  products.  The  primary  product  of  Helix  is  Atlas:  The  Theory  of  Effective  Systems 
Engineers.  This  document  represents  Atlas  1.0-  expected  to  be  mature  enough  for  individuals 
or  organizations  to  use  without  direct  help  from  the  Helix  team.  It  is  a  standalone  document  to 
detail  the  contents  of  Atlas. 

This  document  does  not  contain  all  of  the  research  that  led  to  the  development  of  Atlas  1.0. 
Instead,  the  detailed  research  results  and  how  they  led  to  Atlas  1.0  are  contained  in  the 
companion  Helix  Technical  Report  (SERC-2016-TR-118).  Individuals  or  organizations  that  want 
not  just  to  use  Atlas  but  to  also  understand  the  rationale  and  methodology  behind  its 
development  should  reference  the  Technical  Report.  Several  earlier  published  Helix  papers  and 
technical  reports  are  also  referred  to  throughout  this  report.  The  reader  is  not  expected  to 
read  the  earlier  technical  reports  or  any  of  the  other  Helix  papers  or  reports,  in  order  to 
understand  Atlas  1.0. 

In  addition,  there  are  tools  that  an  individual  or  organization  can  use  to  support  self- 
assessment  using  Atlas.  The  paper-based  tools  are  contained  in  the  Appendices  of  this  report. 
The  team  has  also  developed  more  easily  tailored  Excel-based  tools,  which  can  be  found  on  the 
Helix  page  of  the  SERC  website  (http://www.sercuarc.org/projects/helix/). 

The  relationship  between  Helix,  Atlas,  the  Technical  Reports,  and  the  tools  is  illustrated  in 
Figure  1. 
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Figure  1.  Relationship  between  Helix  and  Atlas 


1.3  Incremental  Development  of  Atlas 

The  Helix  project  used  an  incremental  approach  to  develop  Atlas.  This  approach  was  designed 
to  enable  publication  and  use  of  aspects  of  Atlas  as  they  became  appropriately  mature,  while 
maintaining  the  expectation  that  Atlas  would  become  more  mature  over  time.  The  increments 
were: 

•  Atlas  0.25:  The  first  draft  of  Atlas  based  on  work  done  in  2014  was  published  as  Atlas 
0.25  in  November  2014.  It  included  key  elements  that  explain  the  effectiveness  of 
systems  engineers,  and  a  preliminary  explanation  of  the  relationships  between  those 
elements.  The  structure  and  variables  of  the  proficiency  model  were  also  included, 
along  with  some  initial  analysis  of  career  paths. 

•  Atlas  0.5 :  Based  on  subsequent  work  done  in  2015,  Atlas  0.5  was  published  in 
December  2015.  It  reflected  further  understanding  of  the  elements  of  Atlas  and  their 
inter-relationships.  Significant  new  work  was  done  in  the  area  of  career  paths  and  0.5 
incorporated  initial  efforts  to  use  Atlas  to  assess  the  level  of  proficiency  of  systems 
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engineers.  Atlas  0.5  was  mature  enough  for  an  individual  or  an  organization  to  use  and 
gain  valuable  insights  with  some  guidance  from  the  Helix  team. 

•  Atlas  0.6:  Was  an  incremental  improvement  to  Atlas  0.5.  It  contained  additional  detail 
and  analysis  for  areas  that  were  less  mature  in  0.5,  namely:  mentoring,  personal 
initiatives,  and  organizational  initiatives.  Atlas  0.6  was  not  created  as  a  stand-alone 
document,  but  rather  as  a  supplement  to  0.5. 

•  Atlas  1.0:  Atlas  1.0  included  a  more  complete  description  of  the  elements  of  Atlas  and  their 
inter-relationships.  Atlas  1.0  is  believed  to  be  mature  enough  for  independent  deployment  and 
assessment  by  individuals  and  organizations  with  little  or  no  guidance  from  the  Helix  team.  In 
addition,  the  frameworks  presented  in  Atlas  1.0  have  been  validated  using  data  from  outside 
the  US  DoD,  and  therefore  is  believed  to  be  applicable  to  systems  engineers  in  a  variety  of 
domains.  This  is  intentional.  Though  the  initial  impetus  for  the  work  was  based  on  the  needs  of 
the  US  DoD,  the  Helix  team  believes  that  a  more  generic  framework  which  benefits  all  systems 
engineers,  regardless  of  domain,  is  both  more  beneficial  to  the  community  at  large  and, 
ultimately  will  benefit  the  US  DoD  by  setting  consistent  expectations  for  practitioners  across 
domains. 

•  Atlas  1.1:  This  is  an  incremental  update  to  Atlas  that  reflects  the  teams'  learning  in  2017. 


1.4  AboutThis  Document 

This  technical  report  is  written  as  a  standalone  document,  presenting  version  1.0  of  Atlas:  The 
Theory  of  Effective  Systems  Engineers.  Several  earlier  published  Helix  papers  and  technical 
reports  are  referred  to  throughout  this  report.  However,  the  reader  is  not  required  to  read  the 
earlier  technical  reports  or  any  of  the  other  Helix  papers  or  reports,  in  order  to  understand 
Atlas  1.0. 

Readers  should  note  the  following  about  the  report: 

•  Throughout  the  report,  the  term  'Helix'  is  used  to  denote  either  the  project  or  the  team 
that  performed  the  work  in  developing  Atlas. 

•  The  Guide  to  the  Systems  Engineering  Body  of  Knowledge  (SEBoK)  is  used  across  the 
report  as  the  primary  source  of  consistent  terminology  and  definitions  relevant  to 
systems  engineering.  (BKCASE  Editorial  Board  2016) 

•  All  insights  and  observations  are  presented  only  in  an  anonymous,  aggregated  manner. 
Individuals  or  organizations  that  participated  in  the  Helix  project  are  neither  named  nor 
are  they  identifiable  from  this  report. 

The  report  is  organized  as  follows: 
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2  Methodology 


This  section  provides  detail  on  how  the  Helix  project  has  been  structured  since  its  inception. 
This  includes  overarching  methodology,  details  of  analytic  approaches,  and  detail  about  the 
dataset.  The  organization  of  Section  2  is  as  follows: 

•  The  overarching  philosophical  approaches  to  the  research  (Sections  2. 1-2. 3), 

•  The  detailed  analysis  approaches  for  the  research  (Sections  2.4  and  2.5), 

•  A  summary  of  the  methodology  used  for  a  master's  capstone  project  on  Helix  (2.6), 

•  An  overview  of  the  dataset  (Section  2.7),  and 

•  Guidance  on  how  to  interpret  the  data,  including  limitations  of  the  dataset  (Section  2.8). 


2.1  Research  Philosophy 

Helix  is  primarily  a  qualitative  study,  with  the  primary  means  of  data  collection  being  interviews 
with  systems  engineers.  From  2012-2013,  the  Helix  team  focused  on  a  mixed-methods 
approach  (Creswell  and  Plano  2011),  combining  the  development  of  basic  research  questions 
with  a  grounded  theory  approach.  Grounded  theory  was  developed  in  the  social  sciences  as  a 
method  for  developing  theory  that  is  grounded  in  data  that  is  systematically  gathered  and 
analyzed.  (Goulding  2002)  This  approach  allows  the  data  itself  to  drive  points  of  further  inquiry, 
guide  categorization,  etc.;  rather  than  starting  analysis  with  an  existing  framework,  all  of  the 
data  is  reviewed  holistically  and  any  potential  areas  of  interest  are  coded.  Over  time  patterns 
emerge  and  these  guide  further  data  collection  and  analysis.  The  development  of  driving 
research  questions  makes  the  Helix  project  mixed  method  as  opposed  to  pure  grounded 
theory. 

When  performing  initial  data  coding,  the  Helix  team  coded  all  data,  not  making  suppositions 
about  which  data  would  prove  "important".  The  team  also  compared  data  collected  against  the 
existing  literature  where  possible.  For  example,  as  systems  engineers  defined  the  activities  that 
they  perform,  the  team  collected  and  organized  the  raw  data  but  also  compared  it  to  the 
"Twelve  Roles  of  Systems  Engineers"  defined  in  (Sheard  1996). 

This  approach  still  reflects  the  philosophy  of  Helix:  Atlas  1.0  is  largely  a  reflection  of  the  data, 
using  the  grounded  theory  principles  to  "let  the  data  speak". 
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2.2  Overarching  Research  Methodology 


The  research  methodology  adopted  for  Helix  research  may  be  considered  to  be  a  modified 
grounded  theory  based  approach,  employing  qualitative  and  quantitative  research  methods. 

During  2013  and  2014,  Helix  primarily  focused  on  data  collection  from  DoD  and  DIB 
organizations  through  semi-structured  in-person  interviews  with  individuals  or  small  groups, 
continually  refining  the  interview  questions  and  process.  Follow-up  interviews  were  conducted 
by  telephone  with  most  of  the  participants.  Analysis  of  the  data  to  address  the  Helix  research 
questions  offered  insights  into  the  effectiveness  of  systems  engineers  and  led  to  the 
development  of  an  early  version  of  Atlas  that  was  published  in  November  2014.  During  2015, 
data  collection  was  expanded  to  organizations  outside  of  DoD  and  DIB,  and  Atlas  0.25  was 
validated  and  improved  upon,  leading  to  the  next  version.  Atlas  0.5,  published  in  December 
2015. 

The  Helix  project  adopted  a  grounded  theory  approach  because  it  did  not  presuppose  any 
specific  theory  or  propose  any  hypotheses  at  the  start  of  the  project.  Grounded  theory  was 
developed  in  the  social  sciences  as  a  method  for  developing  theory  that  is  grounded  in  data 
that  is  systematically  gathered  and  analyzed  (Goulding  2002).  Rather  than  beginning  with  a 
hypothesis,  the  first  step  was  data  collection.  This  approach  is  unusual  in  engineering  research, 
where  a  researcher  traditionally  begins  with  a  theoretical  framework  that  he  or  she  applies  to 
the  phenomenon  to  be  studied.  In  the  Helix  project,  the  data  collected  from  the  many  semi- 
structured  interviews  were  marked  up  with  codes  that  were  grouped  into  concepts,  that  led  to 
the  identification  of  constructs  and  categories  that  formed  the  building  blocks  of  Atlas.  This 
approach  minimized  any  bias  that  might  be  introduced  by  the  researchers,  instead  allowing  the 
large  data  set  collected  through  the  Helix  project  to  drive  theory  development.  Having 
established  a  preliminary  theory  of  effective  system  engineers  and  proficiency  model  of 
systems  engineers,  data  collection  and  interviews  conducted  during  2015  focused  on  validating 
Atlas,  and  refining  the  theory  towards  developing  Atlas  1.0  in  2016. 

Qualitative  research  aims  to  create  or  discover  what  things  are  made  of,  and  what  is  created  or 
discovered  are  called  constructs.  Qualitative  research  is  useful  for  obtaining  insight  into 
situations  and  problems  on  which  one  has  little  knowledge  a  priori.  This  method  is  commonly 
used  for  providing  in-depth  descriptions  of  procedures,  beliefs  and  knowledge,  including  the 
opinions  of  respondents  about  particular  issues;  detailed  data  is  gathered  through  open-ended 
questions.  Data  collection  for  the  Helix  project  and  subsequent  analysis  of  the  data  was 
primarily  done  employing  qualitative  research  methods;  appropriate  software  tools  were  used 
to  support  coding  and  identification  of  constructs. 

Quantitative  research  begins  once  initial  constructs  are  in  hand.  It  attempts  to  gather  data  by 
objective  methods  to  provide  information  about  relations,  comparisons,  and  predictions.  In  the 
context  of  the  Helix  project,  quantitative  research  was  performed  once  initial  constructs  for 
demographics  of  systems  engineers,  their  organizations,  and  their  career  paths  were 
established.  Data  was  collected  from  their  resumes,  as  well  as  through  pointed  questions 
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during  interviews.  Quantitative  analysis  continues  to  be  performed  on  various  elements  of  Atlas 
that  were  developed  based  on  qualitative  research,  particularly  on  the  proficiency  model. 


2.3  Helix  Research  Process 

The  Helix  research  methodology  discussed  in  the  preceding  section  was  deployed  using  the 
research  process  illustrated  in  Figure  2  below.  The  Helix  research  process  consists  of  seven 
major  steps: 

A.  Preparation  for  Data  Collection 

B.  Data  Collection 

C.  Data  Analysis 

D.  Methodology  Review 

E.  Theory  Development 

F.  Publishing 

G.  Validation,  Feedback  &  Deployment 


A.  Preparation  For 
Data  Collection 


Al.  Identify 
Organizations  and 
Individuals 


A2.  Provide 
Project  Materials 
to  Participants 


B.  Data  Collection 


Bl.  Collect  Consent 
Forms  &  Resumes 


B2.  Collect  Institutional 
Data 


B3.  Conduct  Initial 
Interviews 


B4.  Conduct  Follow-up 
Interviews 


C.  Data  Analysis 


Cl.  Prepare  Interview 
Summaries 


C2.  Perform 
f  Preliminary  Analysis 


C3.  Perform  Contextual 
Analysis  (Individual  / 
Organizational) 


C4.  Perform  Detailed 
Qualitative  and 
Quantitative  Analysis 


D.  Methodology  Review 

Dl.  Identify  Updates  for 
Interviews 

D2.  Identify  Updates  to 
Institutional  Data  Requests 

D3.  Identify  Updates  for 
Data  Collection  Approach 

\ 


E.  Theory  Development 


El.  Answer  Helix 
Research  Questions 


E2.  Develop  Theory  of 
Systems  Engineers 
(Atlas) 


F.  Publishing 


Fl.  Publish  Helix  Public 
Reports  and  Papers 


F2.  Publish 

Organizational  Profiles 

(Private) 


G.  Validation,  Feedback  & 
Deployment 


Gl.  Validate  Theory  with 
Interview  Participants 


G2.  Deploy  Atlas  with 
Individuals  and 
Organizations 


G3.  Conduct  Helix 
Workshops 


Figure  2.  Helix  Research  Process 
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The  focus  of  Helix  in  2013  was  on  executing  the  loop  A-B-C-D-A  multiple  times  with  different 
organizations.  The  loop  B-C-B  was  executed  a  few  times  when  follow-up  interviews  were 
conducted  with  some  organizations.  During  2014,  in  addition  to  performing  the  A-B-C-D-A  loop 
with  new  organizations,  steps  E-F-G  were  executed  that  led  to  initial  the  development  of  Atlas 
0.25.  In  2015,  much  effort  was  concentrated  in  executing  step  G,  as  well  as  executing  the  loop 
A-B-C-D-A  with  commercial  non-DoD  organizations  as  well  as  with  many  participants  who  did 
not  consider  themselves  to  be  systems  engineers.  In  2016,  the  primary  efforts  were  focused  on 
executing  step  G,  which  included  supporting  organizations  as  they  determined  how  to 
implement  Atlas  and  conducting  extensive  outreach  with  the  systems  engineering  community. 
This  has  led  to  further  refinement  of  Atlas  in  step  E,  leading  to  step  F  -  the  publishing  of  Atlas 
1.0. 

In  2017,  the  Helix  team  executed  loop  A-B-C-D-A  with  three  new  organizations.  The  loop  B-C-B 
was  executed  by  examining  the  existing  dataset  and  comparing  it  to  new  data  collected.  Steps 
E-F-G  were  executed  leading  to  the  updated  to  Atlas  1.1  and  the  development  of  this  report 
and  the  two  companion  documents. 


2.3.1  Preparation  for  Data  Collection  (A) 

Since  Helix  research  is  based  on  a  grounded  theory  approach,  preparation  for  data  collection 
was  the  first  step  executed  in  the  project.  Initially,  organizations  from  within  the  US  DoD  and 
other  organizations  from  the  DIB  were  identified  for  data  collection;  also,  the  primary  focus  was 
on  systems  engineers  in  these  organizations  (Al).  As  Helix  progressed,  in  2015  other 
commercial  organizations  from  non-DoD  sectors  such  as  healthcare  and  information  technology 
were  identified  for  data  collection.  The  latest  reports  and  papers  published  from  the  Helix 
project  were  provided  to  potential  interviewees  (A2).  Based  on  their  willingness  to  participate 
in  Helix  interviews,  the  organization  makes  final  decisions  on  who  participates  in  Helix 
interviews.  In  2016,  the  primary  focus  has  been  on  additional  publication  and  on  assisting  with 
the  implementation  of  Atlas  at  several  organizations.  These  efforts  have  helped  the  team 
understand  where  the  theory  needed  adjustments  to  make  it  easier  to  utilize. 

In  2017,  there  was  a  shift  in  research  questions  from  previous  years  (see  section  3.1.1).  With  a 
different  focus  to  the  research,  the  team  invested  time  in  developing  new  questions  and 
guidelines  for  data  collection  (see  Appendix  B  and  Appendix  C).  The  updated  research  questions 
were  critical  to  enable  the  Helix  team  to  collect  new  data  that  will  bridge  the  previous  results 
with  new  areas  of  inquiry. 


2.3.2  Data  Collection  (B) 

The  first  round  of  data  collection  with  an  organization  is  typically  through  a  site  visit  to  the 
organization,  where  in-person  interviews  are  conducted.  Typically,  there  will  be  2  or  3  Helix 
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interviewers  and  anywhere  from  1  to  6  interviewees  in  a  single  90-minute  interview  session 
(B3).  Following  approved  research  protocols,  a  signed  consent  form  is  collected  from  the 
participants  before  conducting  interviews;  resumes  are  also  requested  from  all  participants 
(Bl).  Any  available  organizational  data  that  will  provide  insights  into  the  systems  engineers  in 
the  organization  and  how  they  are  structured  within  the  organization  are  gathered  before  and 
during  the  site  visit  (B2).  In  2015,  as  the  project  expanded  to  include  non-DoD  participants, 
initial  interviews  were  conducted  over  telephone  when  the  number  of  participants  from  an 
organization  was  very  low.  All  follow-up  interviews  were  conducted  over  telephone  (B4).  In 
2016,  there  have  been  no  additional  interviews.  Instead,  the  team  has  focused  on  analysis  of 
the  data  collected  and  on  assisting  organizations  with  the  implementation  of  Atlas.  Data 
gathered  in  2016  was  focused,  therefore,  on  an  issues  identified  with  implementation  and 
determination  of  whether  those  issues  reflected  a  weakness  in  Atlas  to  be  addressed  or 
whether  they  were  a  reflection  of  the  unique  environment  of  a  given  organization. 

Data  collection  in  2017  was  focused  on  gathering  new  data  to  help  fill  gaps  in  the  existing 
dataset,  focusing  on  organizational  culture,  governance,  and  infrastructure,  and  how  these 
impact  systems  engineers. 


2.3.3  Data  Analysis  (C) 

The  first  step  in  data  analysis  is  to  prepare  summaries  of  all  interview  sessions  (Cl).  Where 
interviewees  permit  audio  recording,  transcripts  are  first  created  then  cleaned  and  prepared  for 
further  analysis.  If  recording  is  not  permitted,  summaries  are  created  from  the  notes  taken 
during  the  interviews.  Preliminary  analysis,  typically  not  employing  significant  effort  on  using 
analysis  tools,  is  performed  to  quickly  identify  additional  questions  to  be  asked  or  additional 
data  to  be  collected  during  follow-up  interviews  (C2).  Since  2014,  significant  research  effort  has 
been  put  into  performing  contextual  analysis  on  an  individual,  particularly  on  her  career  path 
(C3).  Detailed  qualitative  and  quantitative  analyses,  using  software  tools  as  necessary,  have 
been  performed  on  the  large  amounts  of  data  that  have  been  collected  through  Helix 
interviews  (C4).  These  analyses  make  significant  contributions  to  theory  development  efforts 
(F). 


2.3.4  Methodology  Review  (D) 

Data  collection  and  analysis  is  being  performed  iteratively,  as  Helix  continues  to  identify  and 
visit  organizations.  After  any  site  visit  and  before  the  next  one,  a  review  is  conducted  to  identify 
any  updates  to  the  interview  questions  or  process  (Dl).  While  much  organizational  data  is 
desired  for  Helix  analysis,  not  all  information  is  being  made  available  within  an  organization  in  a 
form  that  may  be  readily  shared  with  Helix  researchers.  Based  on  experiences  with 
organizations,  the  nature  and  content  of  organizational  data  requested  has  been  regularly 
updated  (D2).  Based  on  significant  data  analysis  and  theory  development  that  was  performed  in 
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2014,  the  data  collection  approach  was  revised  from  being  a  semi-structured  interview  to  being 
a  discussion  on  assessing  the  proficiencies  of  individuals  and  analyzing  their  career  paths  (D3). 
Feedback  received  from  individuals  and  organizations  on  the  Helix  reports  (G)  also  influenced 
the  updates  performed  in  step  D.  In  2015,  work  on  broad  implementation  has  identified  areas 
has  been  the  focus.  However,  if  additional  in-depth  interviews  are  conducted  in  future,  the 
approach  of  reviewing  findings  and  conducting  interviews  around  existing  frameworks  is 
expected  to  remain  consistent. 


2.3.5  Theory  Development  (E) 

Analysis  performed  on  data  collection  during  2013  focused  primarily  on  answering  the  Helix 
research  questions  at  a  broad  level  (El).  Since  2014,  the  focus  of  analysis  has  been  to  develop 
Atlas  (E2).  Version  0.25  of  Atlas  was  published  in  November  2014.  Atlas  0.5  was  published  in 
December  2015  and  an  incremental  improvement.  Atlas  0.6  was  published  in  April  2016. 
Refining  this  theory,  and  packaging  it  for  independent  assessment  and  deployment  by 
individuals  and  organizations  has  been  the  focus  of  research  efforts  in  2016.  Additional  data 
collected  and  continued  work  with  organizations  implementing  Atlas  culminated  in  the 
development  of  Atlas  1.1. 


2.3.6  Publishing  (F) 

Publishing  reports  and  papers  for  public  consumption  is  a  key  objective  for  Helix  research  (FI). 
All  results  and  observations  reported  in  Helix  publishing  are  done  in  an  anonymous  aggregated 
manner.  Nothing  published  by  Helix  is  traceable  to  any  particular  individual  or  to  an 
organization.  Organizations  may  choose  to  reveal  their  participation  in  the  Helix  project,  but 
they  are  not  listed  in  any  Helix  report.  In  addition,  peer-reviewed  conference  and  journal 
papers  continue  to  be  published  for  wide  dissemination  of  Helix  results.  While  some  form  of  an 
organizational  profile  is  created  as  part  of  internal  Helix  analysis,  in  some  rare  cases,  a  private 
report  is  provided  to  participating  organizations  upon  request  to  support  their  systems 
engineering  workforce  development  efforts  (F2). 

A  complete  list  of  Helix-related  publications  can  be  found  in  Appendix  A. 


2.3.7  Validation,  Feedback  &  Deployment  (G) 

Since  publishing  Atlas  0.5,  step  G  has  become  the  primary  focus  of  the  Helix  process.  In 
implementation  efforts  conducted  in  2016,  the  Atlas  theory  and  proficiency  model  have  been 
validated  in  a  number  of  ways.  During  2015,  Helix  began  deploying  Atlas  with  specific 
organizations  in  an  attempt  to  use  Atlas  to  establish  the  proficiency  levels  and  career  paths  of 
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participants  and  to  be  able  to  discuss  ways  to  develop  their  careers  in  the  future,  towards 
achieving  targeted  levels  of  proficiencies  required  for  particular  senior  positions  within  the 
organization  (Gl).  In  2016,  Helix  helped  a  few  organizations  think  critically  about  how  Atlas 
could  be  implemented  -  including  any  modifications  to  fit  the  organizational  context.  (G2)  The 
result  of  this  work  has  helped  the  Helix  team  to  clearly  identify  where  tailoring  of  Atlas  is 
expected  versus  where  Atlas  is  expected  to  remain  very  consistent  regardless  of  the 
organization,  domain,  etc.  The  primary  expectations  for  tailoring  are  highlighted  in  the 
discussion  of  implications  for  use  of  the  proficiencies. 

The  first  Helix  workshop  was  held  in  July  2014,  with  participation  of  representatives  from  DoD, 
academia,  and  industry,  including  representatives  from  organizations  that  participated  in  Helix 
interviews.  Feedback  from  the  workshop  significantly  shaped  Atlas  0.25.  The  second  Helix 
workshop  was  held  in  August  2015  and  reinforced  the  relevance  and  potential  value  of  Atlas  to 
a  variety  of  systems  engineering  organizations.  The  third  Helix  workshop,  an  early  adopter's 
workshop,  was  held  in  September  2016  and  provided  participants  the  opportunity  to  examine 
Atlas  in  detail  and  even  allowed  participants  the  opportunity  to  use  the  tools.  The  fourth  Helix 
workshop  was  held  in  October  2017  and  focused  on  the  methods  that  had  been  utilized  to  date 
in  employing  Atlas  as  well  as  research  updates  in  2017  (G3). 
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2.4  Qualitative  Analysis 


The  Helix  research  methodology  discussed  in  the  preceding  sections  was  deployed  using  the 
analysis  processes  described  below. 


2.4.1  Coding 

The  interview  dataset  comprises  nearly  6,000  pages  of  transcripts  and  summaries  from  287 
individuals.  In  order  to  make  sense  of  such  a  large  quantity  of  data,  the  Helix  team  uses 
qualitative  data  analysis,  primarily  through  data  coding.  Coding  is  "a  systematic  way  in  which  to 
condense  extensive  data  sets  into  smaller  analyzable  units  through  the  creation  of  categories 
and  concepts  derived  from  the  data."  (Lockyer  2004)  Codes  can  be  layered,  and  evolve  over 
time,  as  explained  below.  When  developing  a  theory,  as  in  the  development  of  Atlas,  categories 
and  codes  are  generated  after  examining  the  collected  data,  aligning  with  the  grounded  theory 
approach.  (Bourque  2004  and  Lockyer  2004)  The  main  type  of  coding  done  by  the  team  so  far  is 
called  "open  coding",  the  purpose  of  which  is  to  break  down,  compare,  and  categorize  data 
(Strauss  and  Corbin  2014). 

The  team  has  used  two  techniques  for  coding:  auto  coding  and  manual  coding.  "Auto  coding"  is 
only  the  first  stage  in  parsing  the  information  contained  in  transcripts  and  is  not  fully 
automated  despite  its  name.  Instead,  as  the  team  reviews  and  cleans  each  transcript,  headings 
are  added  to  the  source  documents  to  block  out  a  large  area  of  text  as  addressing  a  particular 
topic,  such  as  personal  characteristics,  mentoring,  experiences,  etc.  When  the  documents  are 
imported  into  the  qualitative  analysis  tool,  NVIVO,  the  tool  then  automatically  codes  all  text 
under  that  heading  for  the  given  subject.  The  team,  then  can  pull  up  all  auto  coded  text  related 
to  personal  characteristics,  for  example,  from  across  the  entire  data  set  and  examine  it  at  once. 
This  allows  a  more  consistent  look  at  the  related  data  that  can  then  evolve  more  quickly, 
allowing  the  team  to  identify  patterns  that  occur  across  data. 

Auto  coding  is  a  useful  approach,  but  has  its  drawbacks.  One  of  the  strengths  of  the  coding 
approach  is  that  codes  can  overlap  -  individuals  may  discuss  several  issues  together  and 
researchers  can  layer  multiple  codes  together.  Not  only  does  this  help  to  give  a  true 
characterization  of  the  data,  but  common  patterns  in  overlaps  may  provide  useful  insights.  For 
example,  the  proficiency  of  big  picture  thinking  was  often  discussed  simultaneously  with 
several  of  the  values  that  systems  engineers  provide.  This  helped  explain,  for  example,  the 
relationship  between  big  picture  thinking  as  a  critical  skill  and  how  that  approach  can  provide 
value  on  diverse  teams.  But  when  using  auto  coding,  layered  codes  are  not  possible;  in  the 
example  of  big  picture  thinking  and  value,  the  text  would  be  tagged  either  as  "Personal 
Characteristics"  or  "Proficiency"  -  not  both.  Since  auto  coding  is  only  the  first  step,  there  are 
additional  opportunities  to  create  the  layering  and  complexity  that  reflects  the  nature  of  the 
data.  However,  auto  coding  does  limit  the  researcher  to  make  a  choice  about  what  is  most 
important  or  most  prevalent  in  a  section  at  the  outset,  which  raises  the  risk  that  important 
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relationships  could  be  missed  later.  The  other  drawback  to  auto  coding  is  that  categories  had  to 
be  developed  and  applied  to  all  data  and,  therefore,  could  not  happen  early  in  the  project.  The 
team  agreed  to  a  limited  set  of  categories,  largely  aligning  with  elements  of  Atlas  0.5  published 
in  2015. 

If  auto  coding  was  not  used  -  for  example,  if  there  were  a  new  area  of  inquiry,  meaning  that  no 
headings  had  previously  been  identified  and  applied  -  then  the  team  had  to  manually  review 
and  code  all  ~6,000  pages  of  data.  Though  keyword  searches  could  be  used,  there  was  a  risk 
that  data  could  be  missed  if  only  keyword  searches  were  used.  For  this  reason,  the  team  used  a 
variety  of  keyword  searches  related  to  a  given  topic  as  well  as  a  scanning  read  of  a  transcript 
when  doing  the  initial  pass  for  manual  coding.  For  example,  when  looking  for  information  on 
training,  keywords  included,  "train,"  "course,"  "class,"  "learn",  and  "study".  Once  the  initial 
coding  was  complete,  this  is  essentially  equivalent  to  auto  coding  in  terms  of  level  of  depth. 

Additional  codes  were  then  added  to  this  subset  of  the  data  to  further  clarify  the  patterns.  For 
example,  a  total  of  30  individual  personal  characteristics  were  identified  by  participants.  Some 
of  these,  in  the  discussion,  were  directly  linked  to  the  values  that  they  helped  to  provide  - 
these  sections  were  double  coded  for  both  the  characteristic  and  the  value.  Once  all  of  the  data 
had  been  analyzed,  the  team  identified  a  reportable  threshold  -  for  example,  for  personal 
characteristics  there  were  141  excerpts  and  30  characteristics. 

Individual  characteristics  were  mentioned  anywhere  from  15  times  to  only  a  single  time  across 
the  excerpts.  While  none  of  the  characteristics  is  "wrong",  it  was  also  not  useful  to  simply 
provide  a  laundry  list  of  items,  particularly  those  that  were  only  mentioned  once  across  such  a 
large  dataset.  It  was  more  useful  to  first  identify  whether  there  were  any  relationships  between 
items  that  might  help  identify  areas  of  importance.  This  was  done  by  comparing  overlaps 
between  codes.  In  other  words,  a  single  excerpt  might  be  coded  for  multiple  characteristics  that 
were  discussed  together.  By  examining  how  often  characteristics  were  cross-coded,  it  helped  to 
identify  relationships  that  participants  believed  are  important  across  organizations.  For 
example,  in  terms  of  personal  characteristics,  ambition  and  internal  motivation  often  were 
discussed  simultaneously,  which  is  why  they  are  grouped  together  in  Atlas.  Figure  3  provides  an 
example  of  the  coding  comparisons  conducted  by  the  Helix  teams.  The  higher  the  bars,  the 
higher  the  overlap  in  coding  between  characteristics.  This  provides  Helix  with  insight  into 
relationships  between  and  patterns  around  characteristics  based  on  how  interviewees 
discussed  them. 
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Figure  3.  Example  of  Coding  Relationships 

When  reporting,  there  was  also  a  natural  cutoff  -  again,  in  the  example  of  Personal 
Characteristics,  any  characteristic  mentioned  in  more  than  6%  of  the  excerpts.  This  was  a 
natural  threshold  because  the  next  cluster  of  characteristics  occurred  in  approximately  1%  or 
less.  Items  above  the  6%  threshold  were  reported  in  Atlas  0.5 ;  items  below  it  were  not. 
However,  because  the  coding  structure  remains  in  place,  as  additional  data  is  added,  if  other 
characteristics  become  more  prevalent,  then  they  may  be  added  to  the  next  iteration  of  Atlas. 
Personal  Characteristics  were  also  presented  in  the  descending  order  by  the  number  of 
excerpts  in  the  dataset;  e.g.,  "self  awareness"  is  listed  first  because  it  was  discussed  in  the 
highest  number  of  excerpts. 


2.4.2  Tool  Support  and  Combining  Qualitative  Analysis  with  Demographics 

To  support  the  its  analysis,  the  team  has  imported  all  of  the  data  into  NVIVO  (QSR  International 
2016),  a  powerful  and  popular  commercial  qualitative  analysis  tool.  NVIVO  allows  the  team  to 
code  text  as  well  as  overlay  additional  information  and  identifiers  about  the  sources.  This 
replaces  tools  used  early  in  the  project,  namely  a  combination  of  Dedoose  (Dedoose  2016)  and 
Microsoft  Excel,  which  proved  insufficient  to  handle  the  volume  and  diversity  of  data  required. 
In  NVIVO,  the  sources  are  tagged  with  a  code  for  the  organization  in  which  the  individual 
worked  at  the  time  of  the  Helix  interview,  and  then  each  organization  is  linked  to  characteristics 
such  as  whether  it  is  government  or  commercial;  whether  defense,  healthcare,  transportation, 
etc.;  and  how  systems  engineer  are  organized,  e.g.  embedded,  matrixed,  etc.  As  much  as 
possible,  each  comment  is  also  tagged  for  the  individual  who  made  it,  though  in  the  summaries 
from  the  earliest  interviews,  there  are  a  few  exceptions.  Each  individual,  again,  has  several 
characteristics,  including  gender,  whether  they  are  a  systems  engineer  or  a  peer,  and  some 
results  of  the  Helix  career  path  analysis. 
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2.5  Career  Path  Methodology 


In  addition  to  the  analysis  of  interview  data,  the  Helix  team  developed  a  method  for  analyzing 
and  visualizing  the  career  paths  of  systems  engineers.  The  career  path  method  presented  here 
supplements  the  qualitative  data  analysis  described  earlier  with  more  quantitative  information 
about  an  individual's  career.  This  analysis  was  conducted  for  181  systems  engineers1  from  a 
dozen  organizations.  The  initial  data  collection  for  career  analysis  was  conducted  by: 

•  Reviewing  the  resumes  submitted  by  each  individual,  including  chronology, 
organizations,  position  titles,  and  all  descriptive  text  provided  within  the  resumes; 

•  Reviewing  interview  transcripts  and  notes  to  add  detail  to  the  resume  data; 

•  Reviewing  the  preliminary  results  during  follow  up  interviews  to  clarify  analysis. 
Individuals  self-selected  whether  or  not  they  would  like  to  participate  in  follow-up 
interviews;  roughly  half  of  the  individuals  in  the  career  analysis  sample  have 
participated  in  follow-up  interviews;  and 

•  Comparing  the  career  paths  with  existing  Helix  research  on  the  proficiencies  of  systems 
engineers  and  how  career  path  elements  may  relate  to  these  proficiencies.  (Pyster  et  al. 
2014b) 

Using  this  approach,  the  Helix  team  developed  a  method  to  examine  experiences  and  a 
common  framework  to  capture,  analyze,  and  visualize  career  paths.  The  self-assessment  tool(s) 
provided  to  individuals  participating  in  Helix  to  create  their  own  career  paths,  including  their 
proficiencies  over  time,  are  available  in  the  Atlas  1.1. 


2.5.1  Characterizing  a  Systems  Engineer's  Experiences 

Experimental  literature  on  experiences  has  primarily  focused  on  two  metrics  for  experience: 
time  (e.g.  Ford  et  al.  1993;  Schmidt  et  al.  1986;  Firth  1979;  Davidz  2006)  and  the  frequency  of 
times  a  specific  task  or  activity  of  interest  was  performed  (e.g.  Stuart  and  Abetti  2002). 
Additional  literature  classifies  human  subjects  based  on  their  experiences  -  which  is  subtly 
different  than  classifying  the  experiences  themselves  -  often  using  a  combination  of  time  and 
the  frequency  of  tasks  performed.  This  approach  may  also  include  considerations  for  specific 
roles  played  (e.g.  Stuart  and  Abetti  2002,  Kor  2003,  Kirschenbaum  1992).  Additional  literature 
in  the  field  of  systems  engineering,  such  as  Sheard's  "Twelve  Systems  Engineering  Roles"  (1996) 
or  the  Graduate  Reference  Curriculum  for  Systems  Engineering  (GRCSE)  (Pyster  et  al.  2012) 


^he  interviews  for  all  systems  engineers  in  the  Helix  study  are  included  here.  However,  the  resume  data 
was  not  provided  for  some  of  these  individuals,  and  not  all  resume  data  was  sufficient  to  complete  each 
type  of  analysis.  In  general,  the  career  analysis  sample  is  N=1 57.  Where  the  analysis  looks  at  a  subset  of 
the  sample  or  where  individuals  were  eliminated  from  analysis  for  insufficient  data,  the  sample  size  (N-x) 
is  provided  in  the  text. 


Report  No.  SERC-2018-TR-101 


17 


January  16,  2018 


indicate,  though,  that  the  characterization  of  experiences  is  critically  important  to 
understanding  how  experiences  enable  growth. 

The  first  challenge  was  to  determine  a  common  "unit  of  measure"  for  experience.  Though  time 
is  common,  it  was  not  easily  used  in  the  data  available.  For  example,  if  someone  described  a 
position  they  held  over  a  five-year  period,  they  did  not  explain  the  portion  of  time  taken  up  by 
the  activities  they  performed  over  those  five  years.  In  addition,  several  individuals  submitted 
information  on  their  careers  that  included  detailed  descriptions,  but  did  not  include  markers  for 
chronological  time.  Because  of  these  data  limitations,  the  Helix  team  chose  to  use  a  position  as 
the  unit  of  measure  for  experience. 

Based  on  both  the  literature  and  the  Helix  data  itself,  each  position  has  several  characteristics: 

•  Relevance:  A  'relevant'  position  is  one  that  enables  a  systems  engineer  to  develop  the 
proficiencies  critical  to  systems  engineering. 

•  Position:  Every  systems  engineer  who  is  employed  at  an  organization  fills  a  position  that 
is  established  by  the  organization;  that  organization  also  defines  the  roles  and 
responsibilities  to  be  performed.  Helix  considers  position  as  a  'unit  of  measure'  for 
experience,  since  most  of  the  characteristics  of  experience  are  in  the  context  of  the 
position  that  is  held.  A  'systems  engineering'  position  is  one  where  the  individual's 
primary  focus  was  on  systems  engineering  activities. 

•  Chronological  Time:  The  amount  of  time  spent  in  any  particular  position  or  in 
performing  a  role. 

•  Number  of  Organizations:  The  number  of  different  organizations  that  an  individual  has 
worked  at,  not  counting  internal  movement  within  an  organization  across  departments 
or  divisions,  reflects  the  variety  of  types  of  experiences  that  one  may  possess. 

•  Organizational  Sectors:  There  are  many  differences  in  the  general  characteristics  of  an 
organization  based  on  its  sector.  In  Atlas,  three  organizational  sectors  are  identified: 
government,  industry,  and  academia. 

•  Roles:  A  role  is  a  collection  of  related  systems  engineering  activities.  Roles  were 
identified  based  on  the  activities  consistently  performed  by  systems  engineers.  There 
are  16  roles  identified  in  Atlas,  as  described  in  Section  3.5,  below. 

•  Lifecycle  Phases:  Generic  systems  engineering  lifecycle  phases  considered  in  Atlas  are 
based  on  the  lifecycle  phases  in  the  Guide  to  the  Systems  Engineering  Body  of 
Knowledge  (SEBoK).  (BKCASE  Authors  2015) 

•  Systems:  There  are  many  aspects  to  the  types  of  systems  on  which  a  systems  engineer 
could  work.  Working  across  these  different  categories  provides  valuable  experience  to 
an  individual  systems  engineer. 
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o  Domain:  This  is  the  primary  area  of  application  for  the  systems  being  worked  on. 
However,  there  are  many  domain  categorizations;  some  domains  also  relate  to 
industry  sectors. 

o  Type:  Product  systems,  service  systems,  and  enterprise  systems  are  three  major 
types  of  systems,  depending  on  the  nature  and  composition  of  the  system  of 
interest.  System  of  systems  is  another  paradigm  in  systems  engineering,  and 
could  be  a  combination  of  one  or  more  types  of  systems. 

o  Level:  A  systems  engineer  could  work  on  various  levels  of  a  system: 
component/element,  subsystem,  system,  and  platform  or  system  of  systems. 

By  using  the  data  available  for  each  individual,  the  characteristics  of  each  position  played  and 
the  order  that  they  played  them  can  be  identified.  Looking  for  patterns  across  the  Helix  data 
set,  this  information  can  be  used  to  develop  a  preliminary  understanding  of  how  career  paths 
shape  proficiency. 

The  ways  in  which  positions  were  categorized  were  pulled  from  existing  literature  wherever 
possible.  For  example,  a  systems  engineer  working  in  the  commercial  sector  of  a  company  may 
define  life  cycle  in  different  terms  than  those  used  by  a  US  Department  of  Defense  systems 
engineer.  To  normalize  the  discussion,  the  definition  of  life  cycle  stages  from  the  Guide  to  the 
Systems  Engineering  Body  of  Knowledge  (SEBoK)  was  used;  the  interviewee's  own  words  and 
phrasing  were  compared  with  the  descriptions  of  life  cycle  stages  in  the  SEBoK  and  categorized 
appropriately.  (BKCASE  Editorial  Board,  2014)  Likewise,  the  roles  played  by  the  interviewees 
were  based  on  Sarah  Sheard's  "Twelve  Roles  of  Systems  Engineers"  (Sheard  1996),  although 
roles  have  been  added  to  reflect  what  was  seen  in  the  data.  Where  existing  literature  was  not 
available,  categories  were  created  that  reflect  the  character  of  the  data. 


2.5.2  Characterizing  a  Systems  Engineer's  Education 

Education  plays  two  key  roles  in  the  development  of  systems  engineers.  First,  it  provides  the 
foundation  knowledge  to  support  engineering-related  work.  Typically,  this  takes  the  form  of 
undergraduate  education  in  an  engineering  discipline,  technical  field,  or  physical  science. 
Second,  graduate  level  education  is  an  avenue  to  develop  more  advanced  skills,  explore  more 
in-depth  knowledge,  and  help  systems  engineers  grow  as  they  move  through  their  careers. 

The  characterization  of  education  was  much  more  straightforward  than  the  characterization  of 
experiences.  For  each  systems  engineer  in  the  sample,  the  team  recorded: 

•  Chronological  Time:  The  date  of  the  completion  of  the  degree  program. 

•  Type  of  Degree:  This  is  the  level  of  education  an  individual  achieved.  The  categories 
used  were:  bachelor's,  master's,  and  doctor  of  philosophy  (PhD).  For  this  analysis,  only 
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education  that  resulted  in  a  degree  was  recorded.  Individuals  did  receive  graduate 
certificates  or  took  individual  courses,  but  there  was  not  enough  data  to  draw  any 
meaningful  conclusions.  Also,  if  a  degree  was  in  progress  but  not  completed,  it  was  not 
recorded. 

•  Field  of  Study:  The  primary  discipline  on  which  the  individual's  education  was  focused. 
These  were  initially  recorded  as  reported.  Over  time,  categories  of  related  fields  of 
study  were  created. 

All  systems  engineers  in  the  Helix  sample  held  at  least  a  bachelor's  degree  and  the  majority  - 
58%  -  held  at  least  a  master's  degree. 


2.5.3  Identifying  Key  Positions 

A  third  aspect  of  career  paths  are  the  key  milestones  for  a  systems  engineer's  career.  The  Helix 
team  focused  on  major  steps  or  changes  in  a  systems  engineer's  positions.  A  position  is 
equivalent  to  the  roles  and  responsibilities  associated  with  an  individual's  title.  Organizations 
will  define  what  roles  and  responsibilities  each  position  contains  and  position  descriptions  may 
not  translate  across  organizations.  The  key  positions  identified  for  systems  engineer  included: 

•  First  systems  engineering  position.  This  was  self-identified  by  participants  as  the  first 
position  in  which  systems  engineering  responsibilities  were  the  primary  focus  of  a 
position,  though  they  may  have  non-systems  engineering  responsibilities  as  well.  This 
was  often  difficult  to  identify,  because  participants  indicated  that  their  roles  often 
transitioned  gradually  and  it  was  hard  to  identify  when  they  officially  became  systems 
engineers,  especially  because  so  many  never  had  that  specific  title.  The  Helix  team 
recorded  this  information  in  whatever  way  it  was  provided  by  participants.  In  a  few 
organizations,  the  hierarchy  and  structure  for  becoming  a  systems  engineer  was  much 
more  well-defined,  and  for  individuals  in  those  organizations,  the  transition  to  systems 
engineer  was  more  easily  identified. 

•  Chief  systems  engineering  positions.  A  chief  systems  engineer  (CSE)  is  someone  who 
has  formal  responsibility  to  oversee  and  shepherd  the  technical  correctness  of  a  system, 
often  coordinating  with  many  other  systems  engineers  who  have  smaller  scopes  of 
responsibility.  These  milestones  are  any  positions  in  which  an  individual  acted  as  a  CSE, 
regardless  of  their  title  within  their  organization. 

•  Project  manager  positions.  A  project  manager  is  someone  who  has  formal  responsibility 
to  oversee  the  programmatic  aspects  of  a  system,  generally  focused  on  budget  and 
schedule.  Project  management  responsibilities  sometimes  overlap  with  SE 
responsibilities,  particularly  those  around  planning  and  management;  in  some  instances, 
a  CSE  may  also  function  as  a  PM. 
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These  milestones  are  important  for  understanding  how  the  nature  of  a  systems  engineer's  work 
has  changed  over  time.  It  also  gives  insights  into  how  quickly  an  individual  progresses  through 
different  stages  of  her  career.  By  comparing  these  patterns  across  individuals,  common  ranges 
of  progression  can  be  identified,  as  can  outliers.  For  example,  among  the  CSEs  discussed  in  this 
paper,  one  individual  became  a  CSE  only  8  years  after  completing  his  undergraduate  degree. 
However,  only  12%  of  CSEs  gained  their  first  CSE  position  within  10  years  after  entering  the 
workforce;  therefore,  this  is  an  outlier  rather  than  typical  for  CSEs. 


2.5.4  Assessing  Proficiency 

Interviewees  were  asked  about  not  only  their  common  activities  but  what  they  believe  were 
the  critical  knowledge,  skills,  abilities,  behaviors,  and  patterns  of  thought  (cognitions)  that 
enable  them  to  be  effective  in  performing  those  activities.  Helix  calls  these  proficiencies. 

By  coding  all  of  these  responses  individually  and  then  aggregating  like  responses,  the  Helix 
team  has  identified  the  key  proficiencies  of  systems  engineers.  These  are  elaborated  in  (Pyster 
et  al.  2015).  In  brief,  there  are  six  proficiency  areas,  each  of  which  contains  several  related 
groups  of  skills,  or  categories,  as  described  below: 

1.  Math/Science/General  Engineering:  Foundational  concepts  from  mathematics, 
physical  sciences,  and  general  engineering.  Categories  include:  Natural  Science 
Foundations;  Engineering  Fundamentals;  Probability  and  Statistics;  Calculus  & 
Analytical  Geometry;  and  Computing  Fundamentals. 

2.  System's  Domain  &  Operational  Context:  Relevant  domains,  disciplines,  and 
technologies  for  a  given  system  and  its  operation.  Categories  include:  relevant 
domains,  relevant  technologies  and  systems;  relevant  disciplines;  familiarity  with  the 
system's  concept  of  operations. 

3.  Systems  Engineering  Discipline:  Foundation  of  systems  science  and  systems 
engineering  knowledge.  Categories  include:  lifecycle;  systems  engineering 
management;  systems  engineering  methods,  processes,  and  tools;  and  system 
complexity. 

4.  Systems  Engineering  Mindset:  Skills,  behaviors,  and  cognition  associated  with  being 
a  systems  engineer.  Categories  include:  big-picture  thinking;  paradoxical  mindset; 
flexible  comfort  zone;  abstraction;  and  foresight  and  vision. 

5.  Interpersonal  Skills:  Skills  and  behaviors  associated  with  the  ability  to  work 
effectively  in  a  team  environment  and  to  coordinate  across  the  problem  domain  and 
solution  domain.  Categories  include:  communication;  listening  and  comprehension; 
working  in  a  team;  influence,  persuasion,  and  negotiation;  and  building  a  social 
network. 


Report  No.  SERC-2018-TR-101 


21 


January  16,  2018 


6. 


Technical  Leadership:  Skills  and  behaviors  associated  with  the  ability  to  guide  a 
diverse  team  of  experts  toward  a  specific  technical  goal.  Categories  include:  building 
and  orchestrating  a  diverse  team;  balanced  decision  making  and  rational  risk  taking; 
managing  stakeholders  and  their  needs;  conflict  resolution  and  barrier  breaking;  and 
business  and  project  management  skills. 

In  addition,  the  non-systems  engineers  in  the  sample  -  project  managers,  classic  engineers, 
executive  leadership,  and  human  resources  personnel  (HR)  -  were  asked  which  proficiencies 
they  considered  critical  in  the  most  effective  systems  engineers  with  whom  they  worked.  These 
were  also  coded  and  aggregated  with  the  systems  engineers'  responses;  they  validated  the 
existing  categories. 

In  2015,  the  Helix  team  provided  the  and  reviewed  the  draft  proficiency  model  to  participants 
and  had  them  react  to  the  categories  and  structure  directly.  The  existing  structure  was 
validated,  with  no  additional  skills  being  cited  that  did  not  fit  within  existing  categories;  this  did, 
however,  help  the  team  in  re-allocating  some  proficiencies  to  other  categories  to  make  them 
more  easily  understood  by  a  wider  audience. 

Finally,  systems  engineers  were  asked  to  perform  self-assessments  of  their  own  proficiencies  at 
different  points  in  their  careers,  which  could  then  be  overlaid  with  their  career  paths.  Early  on, 
the  Helix  team  would  perform  its  own  assessments  during  these  discussions  and  map  them 
against  the  self-assessments  to  ensure  alignment  between  the  team's  approach  and  the 
participants'.  They  were  also  asked  to  cite  what  they  believed  were  the  most  critical 
proficiencies  for  their  current  positions.  In  addition,  some  were  asked  to  identify  what  they 
believed  were  the  minimum  proficiencies  to  be  effective  in  their  current  positions.  Non-systems 
engineers  also  did  self-assessments,  to  help  identify  where  these  proficiencies  overlap  with 
other  disciplines.  In  addition,  they  were  asked  what  they  believed  were  the  most  critical 
proficiencies  or  the  minimum  proficiency  level  they  would  desire  in  the  systems  engineers  that 
they  work  with.  All  of  this  work  helped  to  validate  the  proficiency  set  as  a  useful  and 
comprehensive  model. 

The  forces  identified  in  Figure  1  -  experiences,  mentoring,  education  and  training  -  are  linked 
to  the  growth  of  proficiency  by  interview  data.  When  an  individual  would  cite  a  critical  skill,  the 
Helix  team  would  ask  how  that  individual  had  developed  that  skill  over  time.  These  types  of 
discussions  were  cross-coded  for  both  the  relevant  force(s)  and  the  related  proficiency(ies). 


2.5.5  Mapping  a  Career  Timeline 

As  described  above,  chronological  time  is  an  attribute  of  all  positions.  The  final  step  in 
developing  a  career  path  map  for  an  individual  was  to  create  a  visualization  over  time  of  all  of 
the  elements  listed  above.  This  visualization  lays  out  all  of  an  individual's  positions,  and  their 
characteristics  over  time,  with  their  education,  the  career  milestones,  and  their  proficiency 
assessments.  Figure  4,  below,  shows  a  generic  example  of  this. 
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In  Figure  4,  only  the  timing,  roles  and  the  lifecycle  stages  characteristics  of  positions  are 
illustrated.  This  is  for  two  reasons:  one  is  ease  of  visualization  in  a  single  graphic  -  though  any 
combination  of  attributes  is  possible  in  this  format  -  and  the  second  is  that  systems  engineers 
were  able  to  provide  the  clearest  discussions  on  how  roles  and  lifecycle  exposure  contribute  to 
proficiency.  For  other  attributes,  these  relationships  were  more  sporadically  represented  in  the 
data;  in  addition,  not  all  systems  engineers  provided  basic  data  on  all  attributes,  but  the  Helix 
team  was  able  to  complete  roles  and  lifecycle  data  for  nearly  the  entire  dataset  (93%  for  roles; 
91%  for  lifecycles). 

By  creating  these  individuals  "maps"  for  each  career  path,  it  is  possible  to  start  identifying 
patterns  -  not  only  in  proficiencies  but  in  the  common  attributes  that  lead  to  similar  proficiency 
profiles.  Additional  analysis  of  the  career  paths  of  individuals  in  similar  roles  was  also  insightful; 
even  though  there  is  some  individuality  to  each  systems  engineer's  career,  the  common 
patterns  indicate  ways  that  systems  engineers  may  typically  grow  -  or  areas  where  certain 
types  of  systems  engineers  differ  from  others.  The  analysis  highlighted  in  this  paper  is  that  of 
chief  systems  engineers  -  not  only  of  their  career  paths  overall,  but  in  a  few  critical  cases, 
highlighting  their  differences  from  other  senior  systems  engineers. 
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Figure  4.  Generic  Career  Path  Mapping 
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2.5.6  Steps  to  Validate  Findings  from  Career  Paths 


There  were  several  ways  in  which  the  findings  from  career  paths  were  verified  and  validated. 
The  most  straightforward  was  verification  through  follow-up  interviews,  where  individuals  were 
presented  with  the  current  analysis  of  their  career  paths  and  asked  to  provide  any  corrections 
or  fill  any  gaps.  Though  only  48%  of  interviewees  also  participated  in  follow-up  interviews,  the 
changes  and  updates  were  minimal,  and  often  reflected  additions  where  certain  aspects  of  an 
individual's  career  had  not  been  discussed  in  the  initial  interviews.  Because  the  reviews  by 
individuals  of  their  own  career  paths  revealed  no  major  issues  with  the  methods,  the  Helix 
team  considered  that  the  method  is  a  valid  approach  to  understanding  the  causes  of  change 
and  growth  over  time. 

In  terms  of  additional  validation,  the  Helix  team  acknowledges  that,  for  a  number  of  reasons, 
the  way  an  individual  progressed  through  her  career  may  not  have  been  an  optimal  approach. 
Participants  were  asked  to  identify  areas  where  what  they  would  have  approached  their 
careers  differently  based  on  their  current  levels  of  insight.  They  were  also  asked  their  general 
satisfaction  with  their  career  progression  and  to  identify  areas  where  colleagues  might  have 
had  a  different,  preferred  approach.  And  because  -  as  explained  in  the  dataset  discussion 
above  -  the  interviewees  identified  themselves  and  their  organizations  identified  them  as  all  of 
"average"  to  "excellent"  effectiveness,  it  is  reasonable  to  draw  conclusions  about  the  career 
paths  of  these  individuals. 

Finally,  in  2015,  the  Helix  team  worked  with  one  organization  to  pilot  the  career  path  approach. 
Individuals  worked  through  an  example  career  path  with  the  Helix  team  and  then  mapped  their 
own  careers  in  real  time.  The  feedback  was  that  this  approach  was  much  better  structured  and 
focused  than  any  career  guidance  they  had  received.  In  some  cases,  this  reinforced  that  their 
current  planning  was  appropriate,  and  other  individuals  reported  that  with  insights  from  their 
career  paths,  they  realized  that  they  needed  to  seek  new  opportunities.  All  of  the  participants 
(n=34)  agreed  that  this  approach  was  useful  and  would  be  a  valuable  tool  for  them  as  individual 
systems  engineers  and  for  the  organization. 


2.6  Capstone  on  Proficiency  and  Career  Paths 

In  2017,  the  Helix  team  worked  with  a  student  at  the  Stevens  Institute  of  Technology,  Matthew 
Partacz.  Mr.  Partacz  joined  the  Helix  team  as  a  student  and  his  capstone  project  utilized 
historical  Helix  data  and  Helix  methodologies,  along  with  additional  approaches,  to  examine  the 
relationship  between  career  paths,  proficiency,  and  project  or  program  performance.  One 
hypothesis  of  Mr.  Partacz's  study  was  that  career  path  has  a  quantifiable  impact  on  an 
individual's  systems  engineering  (SE)  proficiency.  The  following  is  an  excerpt  from  Mr.  Partacz's 
Capstone  Report  (2017),  which  captures  the  strong  correlation  he  found  between  experiences 
and  proficiency. 
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There  ore  many  different  ways  to  look  at  career  path ,  but  for  this  study  career 
path  was  looked  at  very  broadly.  Career  path  was  assessed  by  classifying  each 
individual  in  the  HELIX  dataset  into  one  of  three  categories  as  shown  in  Table  1. 


Table  1:  General  Career  Path  Classification  Definition 


New  Engineer 

Experienced,  Never 
Titled  Systems  Engineer 

Experienced  Titled 
Systems  Engineer 

Years  of 

Less  than  9 

Equal  to  or  greater  than 

Equal  to  or  greater  than  9 

Experience 

years 

9  years 

years 

Positions 

Title's 

0  years  titled  as  Systems 
Engineer 

Greater  than  0  years 
titled  as  Systems 
Engineer 

SE  proficiency ,  as  defined  in  Atlas ,  consists  of  six  different  areas  based  on  HELIX 
interview  data.  Many  individuals  in  the  HELIX  dataset  completed  a  self- 
assessment,  rating  how  they  believe  they  perform  each  SE  proficiency  at  the  time 
of  the  assessment.  The  six  different  areas  are  defined  in  Table  2. 


Table  2:  Atlas  Proficiency  Area  Definitions 


Area 

Definition 

Math/Science/General 

Engineering 

Foundational  concepts  from  mathematics,  physical 
sciences,  and  general  engineering. 

System's  Domain& 
Operational  Context 

Relevant  domains,  disciplines,  and  technologies  for  a 
given  system  and  its  operation. 

Systems  Engineering 

Discipline 

Foundation  of  systems  science  and  systems 
engineering  knowledge. 

Systems  Engineering 

Mindset 

Skills,  behaviors,  and  cognition  associated  with  being 
a  systems  engineer. 

Interpersonal  Skills 

Skills  and  behaviors  associated  with  the  ability  to 
work  effectively  in  a  team  environment  and  to 
coordinate  across  the  problem  domain  and  solution 
domain. 

Technical  Leadership 

Skills  and  behaviors  associated  with  the  ability  to 
guide  a  diverse  team  of  experts  toward  a  specific 
technical  goal. 

With  each  general  career  path  classification  and  proficiency  self-assessment  we 
can  begin  identifying  relationships  between  the  two.  Relationships  are  evaluated 
and  presented  in  two  ways:  (1)  via  mosaic  charts  and  (2)  using  non-parametric 
statistical  analysis,  similarly  to  The  Business  Case  for  Systems  Engineering  Study: 
Results  of  the  Systems  Engineering  Effectiveness  Survey  0,  Section  4.2. 

When  creating  mosaic  charts,  the  only  difference  to  The  Business  Case  for 
Systems  Engineering  Study:  Results  of  the  Systems  Engineering  Effectiveness 
Survey  0,  Section  4.2.1  is  that  career  path  classification  definitions  were 
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arbitrarily  chosen  instead  of  divided  using  the  weighted  summed  indices.  It  is 
also  important  to  note  that  the  weighted  summed  indices  were  determined  using 
the  cumulative  score  for  SE  proficiency  and  not  for  each  individual  area  of  SE 
proficiency. 

Non-parametric  statistical  analysis  was  conducted  just  as  in  The  Business  Case 
for  Systems  Engineering  Study:  Results  of  the  Systems  Engineering  Effectiveness 
Survey  0,  Section  4.2.2.  Goodman  and  Kruskal's  Gamma  and  p-value  is  a 
measure  of  association  which  expresses  the  strength  of  the  relationship  between 
two  ordinal  variables.  Nationally,  Gamma  values  may  be  interpreted  as: 

0.0  <  |  Gamma  |  <  0.2  ->  Weak  relationship 

0.2  <  |  Gamma  |  <  0.3  ->  Moderate  relationship 

0.3  <  |  Gamma  \  <  0.4  ->  Strong  relationship 

0.4  <  |  Gamma  \  ->  Very  strong  relationship 

Important  to  note,  p-values  are  always  cited  with  each  Gamma  statistic.  P- 
values  are  typically  used  for  rejecting  the  null  hypothesis;  the  lower  the  p-value 
the  less  likely  the  magnitude  of  the  relationship  is  to  be  a  chance  occurrence.  It  is 
conventional  that  values  of  p<0.05  or  p<0.01  be  used  as  a  basis  for  rejecting  the 
null  hypothesis. 

For  full  details  on  Mr.  Partacz's  capstone  project,  see  his  full  report  (2017).  With  Mr.  Partacz's 
help  in  data  collection  and  analysis,  the  Helix  team  was  able  to  cross-reference  career  paths 
generated  for  many  of  the  systems  engineers  in  the  sample  with  their  proficiency  self- 
assessments,  a  critical  step  in  further  validating  importance  of  the  career  path  assessment 
approach  and  providing  more  insights  on  how  career  path  patterns  align  with  proficiency.  This 
work  is  reported  in  the  Career  Path  Guidebook. 


2.7  Helix  Data 

Helix  research  uses  a  bottom-up  approach,  based  on  the  data  being  analyzed.  Hence,  it  is 
essential  to  gather  data  that  is  sufficient  in  quantity  and  quality  to  enable  effective 
development  of  Atlas,  and  to  provide  reliable  insights  and  recommendations  that  can  be 
confidently  used  for  the  development  of  effective  systems  engineers. 


2.7.1  Data  Sources 

The  primary  source  of  data  for  Helix  research  is  face-to-face  semi-structured  interviews  with 
participants  at  their  place  of  work.  Additional  information  about  the  participant  and  the 
organization  were  also  collected  as  available.  Another  data  source  that  Helix  gained  access  to 
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was  the  application  data  for  the  INCOSE  Systems  Engineering  Professional  (SEP)  certification 
program. 


2.7. 1.1  Helix  Interview  Data 


From  June  2013,  when  Helix  conducted  its  first  site  visit  for  data  collection,  until  January  2018, 
a  total  of  363  participants  were  interviewed  from  22  organizations.  Typically,  2  to  3  members  of 
the  Helix  team  interviewed  anywhere  from  1  to  6  participants  in  a  single  interview  session. 

Interview  participants,  if  willing,  also  provided  their  resumes  with  details  about  their 
educational  background,  work  experiences,  and  any  other  information  they  wished  to  provide. 

Follow-up  interviews  were  conducted  over  telephone  with  willing  participants,  to  explore  topics 
that  could  not  be  covered  in  the  initial  face-to-face  interviews  or  to  collect  additional 
information  based  on  their  resumes.  Follow-up  interviews  were  also  used  to  validate  results  of 
Helix  analysis. 

In  both  the  initial  interviews  as  well  as  follow-up  interviews,  transcripts  were  created  when 
audio  recording  was  permitted;  when  not  permitted,  summaries  were  prepared  from  notes 
taken  during  the  interviews.  These  transcripts  and  summaries  from  a  total  of  about  270  hours 
of  interviews  form  the  bulk  of  data  that  Helix  analyzed. 


2. 7. 1.2  INCOSE  SEP  Application  Data 

INCOSE  provides  three  different  levels  of  SEP  certification:  Associate  (ASEP),  Certified  (CSEP), 
and  Expert  (ESEP).  Applicants  from  all  over  the  world  seeking  INCOSE  certification  apply  for  the 
appropriate  level  based  on  their  systems  engineering  experiences,  knowledge,  and 
accomplishment.  INCOSE  provided  to  Helix,  under  a  Non-Disclosure  Agreement,  over  3000 
application  forms  received  from  applicants  during  the  period  May  2004  to  May  2014.  Though 
the  application  data  was  available  in  electronic  form,  it  was  not  in  a  format  that  would  readily 
support  analysis.  Significant  time  and  effort  was  spent  in  extraction,  cleaning,  and  tabulating 
the  data  to  enable  further  analysis. 

Analysis  of  INCOSE  data  did  not  directly  contribute  to  the  building  of  Atlas,  but  provided  some 
validation  and  additional  insights  for  the  analysis  of  the  interview  data. 


Report  No.  SERC-2018-TR-101 


27 


January  16,  2018 


2.7.2  Demographics 


2.7.2. 1  Demographics  of  Sample  Population 

Understanding  the  sample  population  is  important,  since  the  interview  data  is  reflective  of  the 
population  from  which  it  has  been  collected,  and  in  turn,  that  data  is  the  basis  for  Atlas.  An 
understanding  of  the  INCOSE  applicants  reveals  the  breadth  of  the  data  that  it  contains. 


2. 7. 2. 2  Demographics  of  Helix  Dataset 

Understanding  the  sample  population  is  important,  since  the  interview  data  is  reflective  of  the 
population  from  which  it  has  been  collected,  and  in  turn,  that  data  is  the  basis  for  identifying 
career  paths.  Following  the  rubric  for  understanding  the  seniority  of  systems  engineers 
presented  in  Figure  5,  the  results  are  presented  in  Figure  5.  Senior  participants  cover  almost 
two-thirds  of  the  population  while  the  remaining  one-third  is  almost  split  almost  evenly 
between  junior  and  mid-Level  participants. 


Seniority  Level  Distribution 

Junior 


Senior 

65% 


Figure  5.  Distribution  of  seniority  levels  across  Helix  dataset 


Figure  6  illustrates  the  distribution  of  participants  based  on  the  "general  career  path 
classifications"  used  in  Partacz  (2017).  It  divides  the  sample  by  individuals  who  are  recognized  - 
and  recognize  themselves  as  systems  engineers  -  and  those  who  do  not.  A  third  category  "new 
engineer"  denotes  any  individual  with  less  than  nine  years  of  experience.  Note  that  this  is 
different  than  the  Helix  seniority  classifications,  which  do  not  depend  on  time. 

It  can  be  observed  that  more  than  two-thirds  of  Helix  participants  are  Experienced  Systems 
Engineers.  New  Engineers  are  slightly  behind  Experienced  Systems  Engineers  with  only  31%  of 
the  participants  being  allocated  to  New  Systems  Engineers.  On  the  other  hand,  an  almost  even 
distribution  occurred  among  Experienced  Systems  Engineers  who  have  never  been  officially 
titled  "systems  engineer"  and  those  who  have. 
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General  Career  Path  Classification  Distribution 


Experienced, 
Titled  Systems 
Engineer,  34% 


Systems 
Engineer,  35% 


Figure  6.  Distribution  of  career  path  classification  across  Helix  dataset  (from  Partacz  2017) 

Figure  7  denotes  the  distribution  of  gender  across  the  Helix  dataset.  It  can  be  observed  that 
more  than  three-fourths  of  participants  are  male  systems  engineers.  In  each  organization,  the 
Helix  team  requested  additional  information  on  the  overarching  systems  engineering  workforce 
-  as  opposed  to  only  the  sampled  individuals.  Most  organizations  could  not  or  chose  not  to 
provide  this  information.  The  Helix  team  does  not  know  if  the  demographics  of  the  sample 
reflect  the  overarching  gender  demographics  of  the  populations  or  is  a  result  of  the  way  in 
which  organizations  selected  individuals  for  participation. 


Participants'  Gender  Distribution 


Figure  7.  Distribution  of  genders  across  Helix  dataset 

To  provide  a  more  detailed  context  about  Helix  findings,  it  is  helpful  to  understand  the  domain 
in  which  the  systems  engineers  interviewed  perform  their  activities.  As  it  can  be  observed  in 
Figure  8,  from  every  four  participants,  three  are  related  to  Defense  type  of  organizations.  The 
rest  of  participants  are  involved  in  the  domains  of  Healthcare,  Transportation, 
Telecommunications,  or  Information  Technology. 
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Individual  by  Organization  Domain 
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Figure  8.  Distribution  of  Individuals  by  Organization  Domain 


Another  classification  of  the  type  of  participant  organizations  is  their  commercial  affiliation. 
Helix  classified  commercial  affiliation  into:  Government,  Industry  and  Federally  Funded 
Research  and  Development  Centers  (FFRDC).  As  it  can  be  observed  in  Figure  9,  more  than  half 
of  the  participants  belong  to  industry  organizations.  The  rest  of  the  dataset  is  distributed 
among  government  entities  and  FFRDC,  the  former  covering  more  than  20%,  while  direct 
government  organizations  slightly  less  than  20%. 


Individuals  by  Organization  Type 


Government  Industry  FFRDC 


Organization  Type 

Figure  9.  Individuals  by  Organization  Type 
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2. 7. 2. 3  Demographics  of  INCOSE  SEP  Applicants 


From  the  over  3000  application  forms,  about  2500  unique  applicants  were  identified  for  further 
analysis.  These  applicants  were  predominantly  from  the  U.S,  but  there  were  others  from  Asia 
and  Europe  as  well,  as  indicated  in  Table  3. 


Table  3.  Geographical  Distribution  of  INCOSE  SEP  Applicants 


Rank 

Country 

#  Applicants 

%  Total 

1 

U.S. 

1847 

74% 

2 

India 

179 

7% 

3 

Germany 

151 

6% 

4 

France 

101 

4% 

5 

U.K. 

49 

2% 

6 

Sweden 

41 

<2% 

7 

Spain 

36 

1% 

Other 

100 

4% 

Information  from  all  the  2504  unique  applicants  was  used  for  analysis  of  education  background; 
a  subset  of  those  applicants  was  analyzed  for  experiences. 


2.8  Interpretation  and  Generalization  Using  the  Dataset 

Helix  is  careful  when  using  the  data  to  understand  whether  and  how  findings  and  conclusions 
about  the  dataset  can  be  generalized  to  the  wider  population  of  systems  engineers.  The  team 
recognizes  that  there  are  some  limitations  based  on  the  sample.  Though  it  is  relatively  large  at 
nearly  300  individuals,  there  is  no  clear  estimate  of  how  many  systems  engineers  are  working  in 
the  US,  let  alone  the  world;  this  makes  it  difficult  to  understand  how  statistically  representative 
the  sample  may  be.  Likewise,  all  interviews  conducted  to  date  have  been  in  the  US;  though  a 
few  individuals  provided  insight  into,  for  example,  education  outside  the  US,  and  some 
organizations  had  units  outside  the  US,  current  findings  reflect  a  US  context.  Likewise,  though 
Helix  has  expanded  beyond  its  initial  defense  roots  to  include  healthcare,  transportation, 
telecommunications,  and  other  industries.  Figure  3  clearly  shows  that  the  majority  of 
individuals  participating  in  the  sample  are  from  the  defense  industry.  Given  these  limitations, 
the  Helix  team  is  careful  not  to  over-interpret  the  data. 

However,  even  with  the  limitations  of  the  sample,  the  team  believes  the  overall  size  of  the 
sample  and  the  diversity  in  terms  of  industries,  organization  types,  and  seniority  makes  any 
findings  and  conclusions  drawn  from  the  data  extremely  useful,  even  if  they  are  not  fully 
generalizable.  The  Helix  team  first  published  Atlas  0.25  in  2014;  since  then  the  coverage  of 
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industries  and  organizations  has  grown  and  the  team  has  nearly  doubled  the  size  of  the  sample. 
And  over  this  time,  there  have  been  some  updates  and  edits,  but  no  major  issues  or  breaks  with 
the  theory  have  been  discovered.  This,  again,  builds  confidence  that  the  existing  sample  is 
sufficient  to  enable  useful  and  insightful  work.  As  the  team  continues  work  on  implementation 
and  application  of  Atlas  (see  Future  Directions),  these  activities  should  generate  greater 
confidence  in  the  generalizability  of  the  data. 
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3  Helix  Activities  in  2017 


This  section  provides  a  brief  overview  of  the  work  of  the  Helix  team  in  2017.  As  cited  below,  the 
primary  results  of  the  study  are  found  in  companion  documents. 


3.1  Work  Focused  on  Effective  Systems  Engineers 

The  Helix  team  completed  conducted  three  site  visits  in  2017,  adding  three  organizations  and 
76  additional  individuals  to  the  dataset  (see  section  2.7  for  a  description  of  the  updated 
dataset).  In  addition,  the  team  performed  additional  analyses  to  create  three  new  documents 
for  the  Helix  library: 

•  Atlas  1.1:  The  Theory  of  Effective  Systems  Engineers  -  (SERC-2018-TR-101-A)  This  is  an 
incremental  evolution  of  Atlas  that  reflects  feedback  from  the  community,  additional 
analysis,  and  maturation  of  the  team's  thinking  in  2017.  In  particular.  Atlas  includes 
minor  updates  on  the  values  systems  engineers  provide,  the  roles  systems  engineers 
play,  the  proficiency  model  for  systems  engineers,  and  the  personal  characteristics  of 
systems  engineers.  Henceforth,  this  will  be  referred  to  as  “Atlas  1.1". 

•  Atlas  Career  Path  Guidebook  -  (SERC-2018-TR-101-B)  This  document  provides  analyses 
of  the  Helix  dataset,  providing  common  patterns  in  systems  engineers'  careers.  The 
Guidebook  also  provides  some  insights  on  questions  commonly  asked  of  the  Helix  team 
around  career  paths  and  the  team's  responses.  Finally,  additional  work  on  linking 
proficiencies  to  career  paths  has  been  completed  and  is  reflected  in  the  guide. 
Henceforth,  this  will  be  referred  to  as  the  “Guidebook" . 

•  Atlas  1.1.  Implementation  Guide:  Moving  from  Theory  Into  Practice  -  (SERC-2018-TR- 
101-C)  Whenever  Atlas  is  presented,  there  are  many  questions  about  how  to  take  the 
theory  and  apply  it  in  practice.  The  Guide  provides  examples  from  organizations  that 
have  implemented  parts  of  Atlas,  and  guidance  created  by  the  Helix  team  based  on 
many  interactions  with  organizations  around  implementation  as  well  as  the  extensive 
Helix  dataset.  Henceforth,  this  will  be  referred  to  as  the  “Implementation  Guide". 
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Figure  10.  Relationship  between  "Helix"  and  "Atlas' 


The  relationships  between  these  documents  are  highlighted  in  Figure  10,  above. 

In  2017,  the  Helix  team  conducted  additional  work  on  effective  systems  engineering 
capabilities,  culminating  in  Atlas  1.1,  which  is  an  incremental  update  reflecting  additional 
analysis  of  existing  data  as  well  as  additional  data  collection  in  2018.  Though  the  changes  in 
Atlas  1.1  are  relatively  minor  (as  reflected  in  the  ".1"  version  number),  they  nevertheless  reflect 
not  only  additional  data  collection  and  analyses,  but  also  incorporate  feedback  from  the 
community.  The  Helix  team  presented  their  work  at  several  community  events,  including  the 
USE  annual  conference,  the  INCOSE  International  Symposium,  the  NDIA  Systems  Engineering 
Conference,  and  the  SERC  Sponsored  Research  Review  (SSRR).  At  each  of  these  events,  the 
team  gained  feedback  from  the  community,  collecting  frequently  asked  questions,  uncovering 
areas  of  confusion,  and  identifying  areas  for  improvement.  The  changes  include: 

•  Revision  of  the  Atlas  1.1  graphic  that  explains  all  of  the  key  variables  included  in  the 
theory.  The  content  did  not  change,  but  the  team  believes  the  update  better  highlights 
the  two  critical  actors  -  individuals  and  organizations. 

•  Reordering  of  the  values  systems  engineers  provide  to  reflect  the  frequency  at  which 
they  occurred  in  the  dataset. 

•  Updating  the  "Requirements  Owner"  and  "Systems  Architect"  roles.  The  activities 
around  functional  architecture  were  moved  from  Requirements  Owner  to  Systems 
Architect  which  both  better  reflect  the  realities  of  the  grouping  of  these  activities  in 
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practice,  but  are  groupings  which  better  align  with  the  mental  models  of  most 
individuals  who  have  engaged  with  the  Helix  team  in  2017. 

•  There  were  several  minor  edits  to  the  proficiency  model.  The  proficiency  areas  stayed 
the  same,  though  the  area  formerly  titled  "Systems  Engineering  Mindset"  is  now 
"Systems  Mindset".  Within  this  area,  the  category  formerly  titled  "flexibility"  has  been 
renamed  "adaptability".  This  not  only  better  reflects  the  comments  in  the  Helix 
interviews  -  which  revolved  around  the  ability  of  an  individual  to  cope  with  a  change  - 
but  also  reduces  confusion.  The  distinction  between  proficiencies  and  personal  enabling 
characteristics  is  nuanced,  and  the  term  "flexibility"  caused  confusion  about  the 
classification  of  the  category.  In  addition,  the  titles  of  categories  in  the  "Technical 
Leadership"  proficiency  area  were  updated  to  increase  clarity.  The  previous  titles 
implied  overlap;  e.g.  "Managing  Stakeholders  with  Diverse  and  Conflicting  Needs"  and 
"Conflict  Resolution  and  Barrier  Breaking"  seemed  to  overlap,  though  their  topics  were 
different.  Though  they  are  related,  they  are  distinct.  The  Helix  team  renamed 
"Managing  Stakeholder  with  Diverse  and  Conflicting  Needs"  to  "Managing  Diverse 
Stakeholders". 

•  There  were  minor  edits  to  personal  characteristics,  particularly  the  update  of  definitions 
for  "inquisitiveness"  and  "life-long  learning"  to  help  clarify  the  distinction  between  the 
two. 

In  addition  to  the  work  on  Atlas  1.1,  the  Helix  team  continued  the  work  of  Dr.  Hutchison, 
building  on  her  dissertation  dataset  on  career  paths  by  creating  career  paths  for  additional 
individuals  in  the  sample  as  well  as  collecting  new  career  path  data.  As  Atlas  was  refined,  the 
team  updated  the  analyses  around  career  paths  to  reflect  these  changes.  The  career  path  data 
was  then  reanalyzed  for  patterns.  The  results  of  these  analyses  can  be  found  in  the  Career  Path 
Guidebook. 

In  addition  to  these  analyses,  the  Guidebook  contains  insights  from  the  team  on  how  to 
interpret  and  apply  some  of  this  guidance.  A  series  of  frequently  asked  questions  -  and  the 
team's  answer  to  these  questions  -  is  incorporated  into  the  guidebook. 

Finally,  Mr.  Matthew  Partacz  based  his  capstone  project  for  his  master's  in  systems 
engieneering  at  Stevens  on  the  Helix  project.  Mr.  Partacz  examined  the  relationship  between 
career  paths  and  proficiency  and  looked  for  a  link  between  these  and  project  performance.  Mr. 
Partacz's  report  can  be  found  on  the  Helix  webpage  (sercuarc.org/projects/helix)  but  key 
findings  related  to  career  path  are  summarized  in  the  Guidebook. 


3.1.1  Additional  Analysis  of  Process  Data 

In  2017,  the  Helix  team  examined  the  existing  data  to  understand  how  the  data  could  support 
this  broader  scope.  The  summary  of  that  work  is  that  the  Helix  team  has  some  information  on 
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Structure,  though  it  varies  widely  between  organizations,  little  on  governance  as  most 
participating  organizations  chose  not  to  share  their  policies  around  systems  engineering  -  or 
simply  did  not  have  such  policies.  There  is  very  little  information  on  the  dataset  on  process.  As 
the  team  was  focused  on  individual  systems  engineers,  process  was  not  the  focus.  In  the  little 
data  available,  there  are  three  findings: 

•  Process  can  be  important,  but  is  not  a  substitute  for  experience.  Overall,  systems 
engineers  who  bring  experience,  insight,  and  foresight  are  valued.  Systems  engineers 
who  only  follow  a  process  are  not. 

•  Inexperienced  systems  engineers  tend  to  lean  very  heavily  on  process  and 
documentation,  which  can  cause  program  mangers  or  peers  to  question  the  values  they 
provide. 

•  Individuals  at  most  organizations  in  the  sample  reported  that  their  employers  reacted  to 
issues  or  failures  by  adding  to  their  processes.  Most  of  these  individuals  complained 
about  the  "bloat"  in  their  processes  as  well  as  the  lack  of  clear  guidance  on  or  authority 
to  tailor  the  processes. 

The  above  are  not  likely  to  be  surprising  to  most  people  who  have  worked  in  or  around  systems 
engineering.  The  Helix  dataset  to  date  primarily  consists  of  these  types  of  qualitative 
assessments,  which  are  useful  but  do  not  contribute  to  the  same  type  of  analyses  as  were 
conducted  on,  for  example,  proficiencies.  The  only  additional  data  are  the  specific  process  an 
organization  might  follow,  such  as  DoD  5000.02.  Going  forward,  the  Helix  team  needs  to  gather 
additional  data  around  the  processes  used  in  organizations  and  how  they  impact  the 
organization's  ability  to  successfully  develop,  maintain,  or  update  systems. 

The  approaches  the  Helix  team  plans  to  use  to  address  these  issues  around  process  in  2018  are 
discussed  in  Section  4,  below. 


3.1.2  Additional  Analysis  of  Organizational  Data 

In  order  to  mine  the  data  for  relevant  attributes  around  organizational  data,  the  team  identified 
and  defined  seven  primary  characteristics  of  organizational  cultures  based  on  work  by  Schein, 
(2010);  Cameron  and  Quinn,  (2011),  and  the  PMBOK  guide  definitions  of  organization  structure. 
See  Appendix  C  for  detailed  definitions  of  the  seven  characteristics. 

1.  Status  afforded  Systems  Engineers  (High,  Low,  In  Transition) 

2.  Structure  (SE  is  Functionally  Centralized  vs.  Distributed  -  Strong,  Balanced,  or  Weak 
Matrix,  Function,  Project) 

3.  Professionalism  (High,  Low,  In  Transition) 

4.  Formality  (High,  Low) 

5.  Influence  (High,  Low) 
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6.  Collaboration  (High,  Low:  Strength,  Breadth) 

7.  Change  (types:  planned,  dynamic;  SE  involvement) 


First,  the  team  selected  four  transcripts  from  four  different  organizations  to  code  for  the  seven 
characteristics  to  a)  determine  how  easy  or  difficult  it  would  be  to  apply  the  rubric  and  b)  get  a 
first  indication  of  the  presence  or  absence  of  interpretable  data  in  the  existing  data  set.  From 
this  analysis  it  was  determined  that  a)  it  was  possible  to  reliably  code  for  the  seven 
characteristics  across  two  coders,  and  b)  in  these  transcripts,  some  culture  indicators  were 
present  while  others  were  not.  In  particular,  SE  status,  structure,  professionalism  and  formality 
were  observed  to  some  degree,  while  there  were  no  clear  indicators  of  influence,  collaboration 
and  SE  involvement  in  organization  change  present  in  this  subsample. 

Second,  the  team  expanded  the  test  of  the  usefulness  of  mining  existing  data  by  coding  the 
seven  characteristics  in  all  transcripts  for  one  organization.  By  examining  325  pages  of 
transcripts  from  33  interviews  from  one  organization,  57  pages  were  extracted  that  contained 
some  reference  to  one  or  more  culture  characteristics. 

Conclusions  from  these  additional  analyses  of  organization  data  include: 

•  The  interviews  contain  some  relevant  data  on  organization  characteristics,  though  for 
several  characteristics  the  data  is  sparse. 

•  Where  it  is  possible  to  code  for  specific  characteristics,  it  is  difficult  to  interpret  the 
meaning  of  the  presence  or  absence  of  specific  characteristics  without  discussion  since 
the  impact  of  culture  characteristics  depends  on  how  members  of  the  culture  interpret 
them.  Meaning,  relevance,  and  impact  must  be  determined  in  context  by  the 
participants  in  the  culture. 

•  These  analyses  enabled  the  team  to  identify  specific  interview  probes  that  can  elicit 
more  meaningful  data  in  future  interviews.  These  probes  were  further  tested  in  the 
three  additional  site  visits  subsequently  performed  in  2017. 

Testing  the  Meaning  and  Value  of  the  Organization  Constructs 


Based  on  the  analyses  of  existing  data  and  experience  with  the  three  new  organizations 
interviewed  in  2017,  the  Helix  team  refined  the  characteristics  and  added  two  additional  areas 
of  interests  to  test  for  relevance  at  the  4th  Annual  Helix  Workshop  in  October  2017.  (See 
Workshop  Report,  Hutchison  et  al.  2017)  Participants  individually  completed  an  analysis  of  their 
own  organizations  based  on  the  seven  culture  characteristics  and  questions  related  to  (a)  how 
the  roles,  status,  and  behaviors  of  the  Non-SE  talent  areas  (ME,  EE,  CS,  etc.)  affect  SE  growth 
and  effectiveness  in  their  organizations,  and  b)  how  the  customer's  (or  customers')  culture, 
values,  and  expectations  of  SE  affect  SE  growth  and  effectiveness  in  their  organizations. 

The  group  discussion  about  the  meaning,  importance  and  applicability  of  the  chosen  cultural 
attributes  to  their  organizations  revealed  that  the  attributes  are  meaningful  and  important  and 
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that  the  presence  and  impact  of  the  attributes  varies  greatly  across  organizations.  Participants 
encouraged  the  team  to  continue  to  explore  these  aspects  of  organization  capability  and  their 
impact  on  SE  growth  and  effectiveness.  The  workshop  validated  that  organizations  can  increase 
their  awareness  about  the  impact  of  their  own  organization  cultures  on  SE  by  using  a  simple 
and  relevant  lens  through  which  to  observe  their  culture  and  behavior. 

Overall,  the  work  in  2017  increases  the  team's  confidence  that: 

1.  Tools  for  observing  and  describing  attributes  of  organization  culture,  structure  and 
governance  can  increase  awareness  and  options  for  growth  within  an  individual 
organization. 

2.  Describing  the  variability  and  inter-relationships  of  various  culture,  governance,  and 
structure  attributes  across  organizations  and  their  impact  on  Systems  Engineering 
capabilities  can  advance  our  understanding  and  development  of  the  Systems 
Engineering  profession. 


3.1.3  Additional  Analysis  of  Methods  and  Tools  Data 


The  Helix  team  reviewed  the  information  in  the  current  dataset  around  systems  engineering 
methods  and  the  tooling  to  support  systems  engineering.  A  list  of  the  most  common  tools 
mentioned  in  the  dataset  can  be  found  in  Table  4. 


Table  4.  Cited  Tools  and  Methods  from  the  Helix  Dataset 


Most  Commonly  Cited 
(>50%  of  Interviews) 


Rational  Rhapsody 
DOORS 


Occassionally  Cited 
(10%  <  x  <  50%) 


MagicDraw 


Rarely  Cited 
(<10%  of  interviews) 


MBSE 
Matlab 
Risk  Radar 
Team  Center 
MKS 
PLM 

3DCADX 
Microsoft  Project 
Microsoft  PowerPoint 
Microsoft  Excel 
SysML 

Artisan  Studio 
Risk  Recon 
SEER 

Vitech  Core 
Red  Mine 
O-NET 
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In  addition  to  examining  the  methods  and  tools  in  the  dataset,  the  Helix  team  also  reviewed 
how  individuals  discussed  these  tools  in  their  interviews.  Because  this  was  not  a  major  focus, 
the  data  is  sparse,  but  the  following  are  common  patterns  seen  to  date: 

•  Individuals  in  almost  every  organization  described  tooling  as  an  issue  or  challenge. 

•  In  some  organizations,  the  individuals  felt  that  the  organization  had  either  not  yet 
identified  sufficient  tools  or  did  not  believe  in  investing  in  tools  specific  to  systems 
engineering. 

•  In  others,  the  tools  had  been  identified  and  were  in  use  in  the  organization,  but 
individuals  did  not  have  access  to  them.  This  was  often  because  there  were  not  enough 
licenses  for  the  number  of  individuals  who  believed  the  tools  would  be  useful  in  their 
work  or  because  those  tools  were  only  utilized  in  some  areas  of  the  organization. 

•  Modeling  was  a  common  point  of  discussion,  particularly  in  terms  of  how  it  could  be 
transformational  to  the  systems  engineering  work.  Different  terms  were  used  to 
describe  this  (model-based  systems  engineering,  model-oriented  systems  engineering, 
model-based  engineering,  etc.),  but  individuals  were  excited  about  the  possibility  of 
using  models  to  help  guide  the  work  and  capture  the  results,  rather  than  the  document- 
based  approach  that  primarily  dominated  the  landscape. 

o  Some  organizations  reported  having  a  model-based  systems  engineering 
initiative.  The  individuals  in  the  organization  were  all  excited  about  this  but 
expressed  that  they  did  not  believe  the  modeling  and  tools  were  sufficiently 
mature  to  support  this  yet.  In  some  organization,  individuals  reported  that  the 
tools  "worked  well  enough"  but  that  the  alignment  between  the  model-based 
approach  and  the  organization's  process  meant  that  significant  level  of  effort 
was  still  required  to  produce  the  expected  document-based  artifacts.  So  while 
the  approach  was  seen  as  beneficial,  it  was  also  considered  a  major  burden  to 
the  programs. 

o  In  one  organization,  an  entire  program  was  developed  using  the  model-based 
approach.  The  benefits  of  this  approach  were  widely  recognized  within  the 
organization,  and  many  individuals  reported  a  desire  to  do  this  approach  on  their 
own  programs.  However,  the  shift  to  a  model-based  approach  in  this 
organization  included  having  several  individuals  dedicated  specifically  to 
modeling,  which  significantly  added  to  costs.  The  organization  and  individuals 
wanted  this  to  broaden,  but  the  funding  challenge  associated  with  this  has  to  be 
resolved. 

The  above  represents  the  current  state  of  the  data  in  Atlas.  Future  work  will  include  examining 
this  as  part  of  the  "infrastructure"  of  the  organization,  which  will  be  described  further  in 
Section  4. 
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3.1.4  Additional  Data  Collection 


In  2017,  the  Helix  team  added  76  new  individuals  and  3  new  organizations  to  the  dataset.  This 
bring  the  dataset  to  a  total  of  363  participants  were  interviewed  from  23  organizations. 

During  the  year  2017,  data  from  3  new  organizations  was  added  into  the  Helix  dataset.  The 
dataset  saw  an  increment  of  76  new  inputs  that  were  analyzed  aggregately  to  identify  career 
paths  in  systems  engineering. 

Figure  11  illustrates  the  comparison  of  seniority  levels  among  Helix  participants.  As  it  can  be 
observed,  the  input  data  is  consistent  with  precious  pattern.  Before  the  year  2017,  the 
demographics  were  the  following:  junior  (19%),  mid-level  (15%)  and  senior  (66%).  Once  the 
information  of  new  participants  was  included,  the  percentage  of  junior  systems  engineers 
decrease  to  (17%).  Mid-level  increased  2%  to  a  total  of  17%.  There  was  no  observed  change 
with  respect  to  the  percentage  of  senior  systems  engineers,  both  classifications  reported  the 
total  demographics  of  systems  engineers  to  be  66%. 


Comparison  of  Seniority  Level 
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Figure  11.  Comparison  of  seniority  level  distribution 

In  regard  to  the  types  of  organization  the  Helix  team  has  collaborated.  Figure  12  denotes  the 
comparison  among  previous  analyzed  data  against  the  new  organizations  included  during  2017. 
Federally  Funded  Research  and  Development  Centers  (FFRDC)type  of  organization  double  in 
percentage  during  the  2017.  Industry  type  of  organizations  decreased  from  65%  to  58%  after 
the  including  the  new  data.  Lastly,  government  entities  were  consistent  with  previous  years 
data. 
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Comparison  of  Organzization  Type 
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Figure  12.  Comparison  of  organization  type  distribution 


3.2  Updates  to  the  Research  Questions 

From  2012  through  2016,  the  Helix  team  focused  on  three  primary  research  questions.  They 
were: 

1.  What  are  the  characteristics  of  systems  engineers? 

2.  What  makes  systems  engineers  effective  and  why? 

3.  How  can  organizations  improve  the  effectiveness  of  their  systems  engineers? 

With  the  publication  of  Atlas  1.0  in  December  2016,  the  Helix  team  believed  that  questions  1 
and  2  were  answered  reasonably  well;  they  could  be  improved  upon,  but  the  foundations  of 
Atlas  had  remained  largely  stable  for  over  a  year  and  additional  data  collected  reinforced  rather 
than  contradicted  the  findings. 

Despite  the  progress  made  to  date,  additional  work  was  required  to  ensure  that  Helix  and  Atlas 
fulfilled  their  potential  impact  with  the  community.  For  example.  Atlas  provides  great  insight 
for  individual  systems  engineers,  but  is  it  possible  to  understand  the  effectiveness  of  systems 
engineers  in  teams  or  a  collective  systems  engineering  workforce.  The  Helix  team  received 
frequent  questions  from  the  community  not  only  about  how  to  implement  Atlas  but  whether 
Helix  would  also  address  the  "next  level"  -  the  broader  context  around  the  systems  engineering 
capabilities  supported  by  the  systems  engineers  in  an  organization. 

With  this  in  mind,  in  2017,  the  research  team  updated  the  research  questions  to  reflect  this 
shift  in  focus.  The  research  questions  became: 
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1.  How  can  organizations  improve  the  effectiveness  of  their  systems  engineering 
workforce?  This  question  has  two  areas  of  consideration:  first,  how  can  the 
effectiveness  of  the  workforce  itself  be  understood?  Is  it  simply  a  collection  of 
individuals  or  are  there  synergies  when  looking  across  the  workforce  as  a  whole?  The 
Helix  team  has  worked  with  several  organizations  to  understand  how  they  can  utilize 
Atlas  in  practice  and  to  date,  each  of  them  treats  their  workforce  as  a  collective  of 
individuals,  not  as  a  holistic  entity.  There  is  additional  research  required  to  better 
understand  whether  this  is  accurate  or  whether  in  systems  thinking  parlance,  "the 
whole  is  truly  greater  than  the  sum  of  its  parts".  This  will  provide  important  insights  as 
organizations  look  to  how  to  grow  their  workforce,  not  just  individual  members  of  it. 

This  leads  to  the  second  part  of  the  question,  which  is  how  can  organizations  actually 
make  an  impact  in  this  area?  In  2017,  the  Helix  team  collected  additional  data  on 
organizational  initiatives,  all  of  which  aligned  with  previous  findings  in  Atlas  1.0. 

2.  How  does  the  effectiveness  of  the  systems  engineering  workforce  impact  the  overall 
systems  engineering  capability  of  an  organization?  The  unspoken  hypothesis  here  is 
that  having  an  effective  systems  engineering  workforce  should  result  in  effective 
systems  engineering  capabilities.  However,  there  are  many  factors  which  impact  a 
systems  engineering  capability  and  while  the  Helix  team  anticipates  that  the  systems 
engineering  workforce  plays  a  critical  role,  as  with  the  Atlas  theory,  the  team  also 
anticipates  that  several  other  factors  will  impact  the  ability  of  even  a  very  skilled 
workforce  to  translate  skill  into  effective  action.  In  order  to  understand  this,  the  team 
has  to  answer  the  third  question: 

3.  What  critical  factors,  in  additional  to  workforce  effectiveness,  are  required  to  enable 
systems  engineering  capability?  Based  on  the  Helix  work  through  2016,  the  team 
developed  an  expected  set  of  variables  that  would  impact  systems  engineering 
capability,  as  illustrated  in  Figure  13. 


Figure  13.  Expected  Variables  to  Impact  Systems  Engineering  Capability 


Report  No.  SERC-2018-TR-101 


42 


January  16,  2018 


The  Helix  team  investigated  potential  uses  of  the  existing  interview  data  to  address  the 
expanded  goal  of  understanding  aspects  of  organizations  that  support  effective  systems 
engineering  capabilities.  Although  the  interviews  were  primarily  focused  on  the  individual,  it 
was  recognized  that  many  interviewees  referred  to  aspects  of  organization  culture,  structure, 
and  governance  when  describing  their  own  growth  and  effectiveness  as  systems  engineers. 


3.3  Related  Work 

This  section  highlights  additional  work  done  on  Helix  in  2017,  specifically  work  by  the  SERC  on  a 
Helix  tool  and  a  book  written  by  Art  Pyster,  Nicole  Hutchison,  and  Deva  Henry: 

•  The  SERC  believes  that  Helix  and  Atlas  provide  critical  insights  for  individuals  who  wish 
to  understand  their  systems  engineering  proficiencies,  their  career  paths,  and  how  the 
two  link  together.  In  2016,  the  Helix  team  published  an  Excel  template  to  facilitate  self- 
assessments  of  both  proficiency  and  career  path,  as  well  as  paper-based  tools.  However, 
for  wide-scale  use  of  Atlas,  the  SERC  believed  that  a  web-based  tool  would  be  more 
critical.  The  SERC  invested  resources  in  2017  to  build  a  tool  to  support  self-assessment. 
At  the  time  of  this  report,  the  first  iteration  of  the  tool  is  nearly  complete  and  will 
provide  the  following  capabilities: 

o  Individual  proficiency  self-assessments,  allowing  multiple  assessments  covering 
many  points  in  time,  including  prompting  users  to  create  a  "target"  proficiency 
profile,  and  enabling  tailoring  of  the  proficiency  model  to  fir  the  individual's 
working  environment; 

o  Proficiency  graphics  to  allow  individuals  to  easily  compare  proficiencies  over 
time,  and 

o  Individual  career  path  self-assessments,  enabling  data  collection  and 
characterization  on  the  critical  variables  of  career  paths  identified  in  Atlas. 

Future  iterations  are  planned  to  enable  visualizations  of  career  paths  over  time  as  well 
as  options  for  organizations,  which  would  enable  them  to  create  an  organization- 
specific  tailoring  of  the  proficiency  model  and  collect  data  across  their  systems 
engineering  workforce. 

•  In  2017,  Art  Pyster,  the  former  PI  for  the  Helix  project,  Nicole  Hutchison,  the  current  PI 
for  the  project,  and  Deva  Henry,  a  researcher  who  worked  on  the  project  almost 
continuously  for  years,  developed  a  proposal  for  a  book  based  around  the  Helix  findings. 
This  book,  which  heavily  references  the  published  Helix  reports,  also  provides  the 
insights  and  opinions  of  the  authors  based  on  their  long  history  with  the  research  as 
well  as  additional  on-the-record  interviews,  case  studies,  and  views  on  the  future  of 
systems  engineering  and  systems  engineers. 
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This  book  will  be  submitted  in  January  2018  and  it  is  anticipated  that  it  will  be  published 
in  late  2018. 
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4  Future  Directions:  Helix  in  2018  and  Beyond 


As  part  of  understanding  what  makes  systems  engineers  effective,  the  Helix  team  has  collected 
data  on  the  ways  that  organizations  try  to  improve  their  systems  engineers'  proficiency  (called 
organizational  development  initiatives)  and  some  basic  trends  have  been  identified.  This  area 
requires  further  study  and  is  critical  to  ensure  that  organizations  that  identify  issues  in  their 
systems  engineering  workforce  or  organizational  support  for  systems  engineering  can 
understand  the  potential  impact  of  and  options  for  improvement. 

Atlas  1.1  outlines  what  impedes  and  enables  systems  engineers,  from  aspects  of  the  individuals 
themselves  -  such  as  proficiency,  career  path,  roles,  and  positions  -  to  aspects  of  the 
organization,  such  as  culture  and  the  specific  ways  that  systems  engineering  is  valued  and 
promoted  (or  not).  The  information  on  this  context  for  systems  engineers  in  the  current  dataset 
is  helpful,  in  particular  in  highlighting  where  these  areas  can  inhibit  systems  engineers' 
effectiveness.  However,  the  dataset  is  not  as  robust  as  is  required  in  this  area  if  organizations 
want  to  use  Atlas  to  make  real  changes  to  improve  their  systems  engineering  workforce. 
Additional  data  must  be  collected  on  organizational  culture,  governance  (including  processes) 
and  structure  (including  infrastructure  and  support  for  systems  engineering). 

The  Helix  team,  in  order  to  draw  clear  conclusions  around  the  organizational  characteristics, 
will  need  to  greatly  increase  the  organizational  variability  of  the  dataset.  This  will  require  a 
change  in  data  collection  approach.  While  some  site  visits  with  large  numbers  of  individuals 
may  still  be  necessary,  the  Helix  team  will  also  need  to  create  a  research  protocol  that  will  allow 
them  to  collect  targeted  organizational  data  with  a  lower  level  of  effort  to  increase  the  number 
of  participating  organizations. 

The  Helix  team,  in  order  to  better  understand  workforce  versus  individual  effectiveness,  must 
add  new  data  on  individual  systems  engineers  and  collect  aggregate  data  on  the  systems 
engineering  workforce  from  additional  organizations.  In  addition,  new  analysis  needs  to  be 
conducted  to  determine  how  to  treat  "the  systems  engineering  workforce"  as  distinct  from 
"individual  systems  engineers." 

In  2017,  the  Systems  Engineering  Research  Center  developed  tooling  to  support  individual  self- 
assessments  based  on  the  Atlas  findings.  The  team  needs  to  evolve  the  tooling  to  allow 
increased  capability,  including  creating  an  option  for  organizations  to  tailor  this  for  their  own 
employees  and  see  summaries  of  the  results.  This  will  aid  the  team  in  increasing  collection  of 
data  that  enables  the  aggregate  analysis  of  individuals  as  a  workforce. 

The  Helix  team  has  begun  exploring  automated  methods  to  glean  additional  insights  from  the 
existing  dataset.  The  team  needs  to  continue  work  on  this  in  2017  to  reduce  the  burden  of 
manual  coding  analysis  on  the  more  than  6,000  pages  of  text. 

The  Helix  team  will  also  conduct  an  extensive  literature  review  of  systems  engineering 
capability  and  build  on  this  work  to  develop  clear  methods  for  understanding  and  assessing 
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how  systems  engineers  are  supported  or  impeded  in  providing  an  effective  systems  engineering 
capability. 

In  addition  to  the  tasks  above,  the  Helix  team  will  begin  building  models  to  support  this  work. 
This  will  include  modeling  tools  that  will  help  enable  organizations  to  assessment  their  current 
level  of  support  for  systems  engineering  and  systems  engineers.  These  models  should 
eventually  be  able  to  help  organizations  explore  the  changes  they  could  make  to  improve  the 
efficacy  of  their  systems  engineering  workforce  and  increase  their  systems  engineering 
capability.  (It  is  likely  that  this  work  will  continue  in  2019.) 

As  the  researchers  continue  data  collection  and  expand  the  variety  of  organizations 
participating,  the  team  will  gather  additional  data  on  the  efficacy  of  organizational 
development  initiatives  that  are  intended  to  grow  systems  engineers. 

The  team  shall  develop  a  method  for  assessing  the  effectiveness  of  systems  engineering 
capability  in  relation  to  the  characteristics  of  the  organization  and  the  systems  engineering 
workforce  and  validate  this  through  data  collection  at  a  variety  of  organizations. 

The  researchers  would  aim  to  develop  five  items  to  support  this  work  in  2018: 

1)  An  initial  framework  for  understanding  the  effectiveness  of  systems  engineering 
(nominally  Atlas0RG  0.5); 

2)  A  set  of  tools  to  support  broader  use  by  both  individuals  and  organizations; 

3)  A  series  of  draft  models  to  support  exploration  of  systems  engineering  capability  with 
organizations;  and 

4)  A  technical  report  providing  the  data  and  analysis  supporting  each  of  the  three  items 
above. 

5)  Draft  models  developed  to  support  the  research. 


4.1  Envisioned  End  State  for  Helix 

The  end  goal  for  Helix,  which  will  likely  be  completed  in  2019,  is  to  develop  a  theory  of  effective 
systems  engineering  capability  -  predicated  on  the  hypothesis  that  if  an  organization  has  a 
sufficiently  skilled  systems  engineering  workforce  and  the  organizational  characteristics  such  as 
culture,  governance,  and  infrastructure  align  with  systems  engineering  -  along  with  the  tools  to 
support  organizations'  self-assessment  of  their  capabilities  and  their  ability  to  change  those 
capabilities.  An  appropriate  analogy  here  is  the  "policy  flight  simulator",  which  according  to 
Rouse  is  "designed  for  the  purpose  of  exploring  alternative  management  policies  at  levels 
ranging  from  individual  organizations  to  national  strategy."  (2014)  As  Rouse  explains,  "the  idea 
is  for  organizational  leaders  to  be  able  to  interactively  explore  alternative  organizational 


Report  No.  SERC-2018-TR-101 


46 


January  16,  2018 


designs  computationally  rather  than  physically.  Such  explorations  allow  rapid  consideration  of 
many  alternatives,  perhaps  as  a  key  step  in  developing  a  vision  for  transforming  an  enterprise." 

By  developing  a  theory  of  what  enables  an  organization  to  have  an  effective  systems 
engineering  capability  (e.g.  Atlas0RG),  the  Helix  team  will  identify  the  key  factors  that  impact  this 
capability  and  the  relationships  between  them.  The  founding  hypothesis  for  this  work  will  be, 
"If  an  organization  has  an  appropriately  sized  and  skilled  systems  engineering  workforce  and 
the  organizational  characteristics  are  supportive  of  systems  engineering,  then  the  organization 
will  have  an  effective  systems  engineering  capability." 

With  this  theory  and  the  rich  dataset  behind  it,  the  team  will  be  able  to  build  tools  to  support 
data  collection  around  critical  variables,  and  models  to  support  this  policy  flight  simulator 
approach.  With  it,  organizations  can  assess  their  current  capability  and  the  factors  that 
influence  it  and  run  simulations  to  determine  what  organizational  changes  may  enable  an 
increased  systems  engineering  capability. 

Though  process  is  anticipated  to  be  one  of  the  variables  that  Helix  will  study,  this  work  is  not 
intended  to  focus  exclusively  on  making  'better  processes'.  There  are  already  approaches  for 
process  improvement  such  as  CMMI,  SixSigma,  and  statistical  process  control.  While  there  may 
not  be  many  examples  of  these  techniques  applied  to  systems  engineering  processes,  the  Helix 
team  nonetheless  does  not  desire  to  create  a  process  improvement  methodology.  Instead, 
areas  of  "process  improvement"  would  include  better  aligning  process  with  other  variables  in 
the  model,  such  as  culture,  governance,  or  workforce  capabilities. 
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5  Conclusion 


The  work  on  Helix  in  2017  was  critical  preparation  for  the  Helix  work  to  come  and  has  furthered 
the  development  of  a  community  of  interest  in  understanding  and  building  exceptional  systems 
engineers  and  systems  engineering  capabilities. 

Accomplishments  that  provide  a  clear  foundation  for  the  next  stages  in  this  research  include: 

•  The  work  performed  mapping  general  theoretical  constructs  about  organizational 
culture  and  governance  to  the  systems  engineering  organizational  context  provides  a 
starting  point  for  designing  future  data  collection,  analysis  strategies,  and  modeling 
approaches. 

•  By  examining  additional  areas  of  the  existing  dataset,  the  team  has  a  clear 
understanding  of  the  status  of  the  current  data  and  where  additional  data  collection  is 
required. 

•  In  updating  the  research  methods  and  interview  questions,  testing  these  with  three 
additional  site  visits,  and  reviewing  the  results,  the  team  believes  it  has  appropriate 
ways  of  asking  for  the  required  additional  data. 

The  research  process  itself  is  developing  a  community  of  professionals  who  can  influence  the 
direction  and  effectiveness  of  systems  engineering  in  diverse  organizations  in  the  future. 

•  By  continuing  the  dialogue  about  systems  engineering  career  paths,  success  stories,  and 
applications  of  the  prior  research,  and  through  the  team's  evolving  understanding  of 
critical  culture  and  governance  attributes.  Helix  is  building  a  community  of  collaborators 
who  are  evolving  the  systems  engineering  profession  in  parallel  with  the  Helix  work. 

•  The  methods  the  team  uses  to  generate  this  dialogue  include  interview  questions  that 
spark  self-awareness,  site-visit  summaries  that  provide  organization  leaders  with  new 
perspectives,  cross-industry  workshops,  conference  presentations,  documents  and 
references. 

Ultimately,  just  as  the  growth  and  effectiveness  of  individual  systems  engineers  depend  upon 
the  commitment  of  an  organization  to  growing  its  systems  engineering  workforce  capability, 
the  value  of  the  Helix  work  beyond  supporting  the  development  of  individual  systems  engineers 
will  be  best  realized  through  the  engagement  of  our  wider  community  of  leaders  of  the  systems 
engineering  profession.  The  Helix  team  is  committed  to  continuing  this  dialogue  in  2018  and 
beyond. 
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Appendix  B:  Updated  Interview  Questions  for  Helix 


Individually  Focused 

Getting  to  Know  You 

IND01  How  did  you  get  in  to  systems  engineering? 

IND02  Do  you  consider  yourself  a  systems  engineer? 

IND03  Define  your  current  position. 

IND04  How  would  you  rate  your  own  clarity  about  your  role  as  a  systems  engineer  right  now 
-  high,  medium,  or  low  clarity?  Why? 

Exploring  Organizational  Characteristics 

GRP01  How  do  systems  engineers  integrate  into  teams  here?  (What  is  the  day  in  the  life  of  a 
systems  engineer  like  with  respect  to  who  they  work  with  and  what  they  do 
together?) 

GRP02  How  do  you  think  others  on  your  team  would  rate  the  clarity  of  your  role  as  a  systems 
engineer  right  now  -  high,  medium,  or  low  clarity?  Why? 

GRP03  We  are  going  to  do  some  free  word  associations.  This  is  when  I  say  a  word  or  phrase 
and  you  write  down,  without  filter,  the  first  three  things  that  come  to  mind.  Let's 
practice  aloud:  apple;  car.  OK,  so  thinking  about  your  experience  in  this  organization, 
please  write  down  the  first  three  words  that  come  to  mind  when  I  say  each  phrase 
[worksheet  for  participants]:  systems  engineering;  systems  engineering  management; 
senior  leadership;  [organization  name], 

GRP04  Draw  a  picture  that  shows  how  systems  engineering  is  really  done  here,  in  your 
experience. 

(Follow  up:  Thinking  about  the  culture  of  the  organization,  what  aspects  of  "the  way 
we  do  things  around  here"  contribute  to  SE  growth  and  success  and  which  hinder  SE 
growth  and  success  here?) 

GRP05  How  would  you  describe  the  status  of  systems  engineering  in  this  organization?  What 
is  your  rationale?  (Follow  up:  Are  systems  engineers  considered  leaders?) 

GRP06  What  kinds  of  decisions  are  highly  influenced  or  "owned"  by  systems  engineers  in  this 
company? 

(Follow  up:  Are  there  any  kinds  of  decisions  where  you  are  not  at  the  table  and  think 
systems  engineers  should  be  involved?) 

GRP07  When  you  think  of  systems  engineering  as  a  profession,  would  you  say  you  feel  very 
connected,  moderately  connected,  slightly  connected  or  not  connected  to  the  broader 
systems  engineering  community?  Why  do  you  say  that? 

GRP08  What's  changing  in  your  organization  and  what  role  do  SE's  have  in  driving  those 
changes? 

GRP09  How  do  you  think  that  the  nature  of  the  work  you  do  influences  your  approach  to 
systems  engineering? 

GRP10  How  do  projects  get  staffed? 

Organizational  Change 
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ORG08  If  you  could  change  one  thing  in  this  organization  that  would  improve  your  systems 
engineering  capabilities,  what  would  it  be?  [Note:  time/money/power  are  not  an 
issue.] 


Organizationally  Focused 

ORGOl  What  is  Systems  Engineering? 

ORG02  What  does  the  organization  value  about  systems  engineering? 

ORG03  Where  does  systems  engineering  fit  in  the  organizational  structure? 

ORG04  How  does  the  organization  expect  systems  engineers  to  be  integrated  into  its 
engineering  teams? 

ORG05  What  is  the  management  style  in  the  organization?  How  is  the  SE  workforce  included  in 
the  management  process? 

ORG06  What  practices  does  the  organization  currently  implement  to  minimize  systems 
engineering  turnover?  (i.e.  retention  incentives,  career  development  options) 

ORG07  Is  there  a  gap  between  the  skills  of  your  systems  engineering  workforce  and  your 
organizational  need?  (Follow  up:  What  are  you  doing  to  fill  that  gap?) 

ORG08  If  you  could  change  one  thing  in  this  organization  that  would  improve  your  systems 
engineering  capabilities,  what  would  it  be?  [Note:  time/money/power  are  not  an 
issue.] 
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Appendix  C:  Cultural  Characteristics  -  Definitions  and  Related  Questions 


1.  Status  afforded  Systems  Engineers  (High,  Low,  In  Transition) 

•  Perceived  by  SEs 

•  Espoused  as  valued  by  leadership  in  Annual  Reports  and  company  docs 

•  Formal  availability  and  actual  use  of  systems  engineering  rewards 

•  Examples  of  informal  recognition 

•  Key  leadership  roles  filled  by  Systems  Engineers 

2.  Structure  (SE  Functionally  Centralized  vs.  Distributed  -  Matrix,  Function,  Project) 

•  Evidence  of  managing  the  polarities  of  both  functional  and  distributed  structures: 
strong  functional  support  (training,  mentoring,  career  ladders)  and  strong  cross¬ 
functional  collaboration  (integration  in  project  teams,  cross-business  contributions) 

Strong  Matrix  Organization  Structure 

•  In  strong  matrix  organizations,  most  of  the  power  and  authority  is  held  by  project 
managers.  Project  managers  have  a  full  time  role,  have  a  full  time  project  management 
administrative  staff  under  them  and  control  the  project  budget.  The  strong  matrix 
structure  has  a  lot  of  the  characteristics  of  a  "projectized"  organization. 

•  The  functional  manager  will  have  a  very  limited  role  within  the  Strong  Matrix 
Organization. 

Balanced  Matrix  Organization  Structure 

•  In  balanced  matrix  organizations,  power  and  authority  are  shared  between  the 
functional  managers  and  the  project  managers.  Although  project  managers  have  a  full 
time  role,  they  have  a  part  time  or  otherwise  limited  project  management 
administrative  staff  under  them.  In  this  type  of  structure,  both  managers  control  the 
project  budget. 

Weak  Matrix  Organization  Structure 

•  In  weak  matrix  organizations,  project  managers  will  have  limited  power  and  authority. 
They  will  have  a  part  time  role  and  no  administrative  staff  will  report  to  them.  Their  role 
will  be  more  like  a  coordinator  or  an  expediter.  Here,  the  functional  manager  controls 
the  project  budget. 

•  A  weak  matrix  organization  structure  resembles  the  characteristics  of  a  functional 
organization  structure. 

3.  Professionalism  (High,  Low,  In  Transition) 

•  Self-Regard:  self-reported  pride  in  functional  expertise  and  contributions,  participation  in 

industry  SE  groups 

•  Other-Regard:  encouraged  to  submit  papers  and  participate  in  industry  groups  by 

leadership,  budget  for  professional  activities 

•  Other-Regard:  Viewed  by  non-systems  engineering  colleagues  as  acting  in  integrity  with 

systems  engineering  values  and  ethics 

4.  Role  Formality  (High,  Low) 
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•  SEs  do  prescribed  job  applying  prescribed  skills  vs.  SEs  fulfill  multiple  roles,  some  of  which 
are  not  related  to  systems  engineering 

•  Degree  of  specificity  of  roles 

•  Degree  of  shared  understanding  or  clarity  about  roles 

•  Types  of  SE  training,  rotations,  mentorships,  apprenticeships,  ladder,  titles,  hiring 


5.  Influence  (High,  Low) 

•  Reach  (narrow  -  broad) 

•  Within  team,  cross-functional,  cross  level,  cross  company 

•  Decision  Ownership 

•  Which  decisions  do  they  own?  Which  do  they  contribute  to?  From  which  are 
they  excluded? 

6.  Collaboration  (High,  Low:  Strength,  Breadth) 

•  Team  participation 

•  percent  of  time?  co-location/ distance  collaboration:  time/support  for 

•  act  as  team  leaders,  members  or  "SME  consultants"  (structure?) 

•  Types  of  cross-functional  daily  work 

•  Types  of  joint  learning,  e.g.,  participation  in  after-action  reviews 

•  Diversity  of  team  membership  (functions,  levels,  customer  involvement...) 

•  Self-described  and  company-espoused  values  re:  teamwork,  joint  problem  solving  and 
conflict  resolution,  group  creativity  and  innovation 

7.  Change 

•  What  formal  initiatives  are  ongoing? 

•  Who  sponsors  them?  Who  participates? 

•  Roles  of  SEs  and  SE  Leaders  in  the  change? 

•  Links  to  corporate  or  SE  strategies? 

•  What  organization,  function,  culture  or  process  change  initiatives  do  SEs  sponsor  or 
lead? 

•  Informal  change 

•  Do  SEs  and  Non-SEs  view  SEs  as  champions  for  change  (possible  areas:  best 
practices,  teaming,  communication,  technical  innovation...) 
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